Front cover

Managing FlashSystem
Performance with IBM
Spectrum Control
Marion Hejny
Falk Schneider
Bert Dufrasne

Redpaper
International Technical Support Organization

Managing FlashSystem Performance with IBM

Spectrum Control

May 2018

REDP-5381-00
 Note: Before using this information and the product it supports, read the information in “Notices” on
page vii.

First Edition (May 2018)

This edition applies to IBM Spectrum Control Version 5.2.14.

This document was created or updated on May 3, 2018.

© Copyright International Business Machines Corporation 2018. All rights reserved.

Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule
Contract with IBM Corp.
Contents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .v
Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .v
Now you can become a published author, too! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi
Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi
Stay connected to IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi

Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii

Chapter 1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 IBM FlashSystem products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 IBM Spectrum Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Chapter 2. Viewing performance information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1 Prerequisites for monitoring FlashSystem devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.1 Adding FlashSystem devices to IBM Spectrum Control . . . . . . . . . . . . . . . . . . . . . 6
2.1.2 Considerations for Performance Monitoring with IBM Spectrum Control . . . . . . . 10
2.1.3 Special considerations for FlashSystem devices . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2 Monitoring performance for FlashSystem 840/ FlashSystem 900. . . . . . . . . . . . . . . . . 17
2.2.1 Monitoring FlashSystem 840 an FlashSystem 900 using native tools . . . . . . . . . 17
2.2.2 Monitoring FlashSystem 840 / 900 with IBM Spectrum Control . . . . . . . . . . . . . . 24
2.2.3 Performance metrics for IBM FlashSystem 840 / 900. . . . . . . . . . . . . . . . . . . . . . 28
2.2.4 Comparing native GUI and Spectrum Control. . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.3 Monitoring Performance for FlashSystem V9000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.3.1 Monitoring FlashSystem V9000 with the native tools . . . . . . . . . . . . . . . . . . . . . . 37
2.3.2 Monitoring V9000 with IBM Spectrum Control . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.3.3 Comparing V9000 native tools with IBM Spectrum Control . . . . . . . . . . . . . . . . . 45
2.4 Monitoring performance for FlashSystem A9000 or A9000R . . . . . . . . . . . . . . . . . . . . 46
2.4.1 Monitoring A9000 or A9000R with HyperScale Manager . . . . . . . . . . . . . . . . . . . 46
2.4.2 Monitoring A9000 with IBM Spectrum Control . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.4.3 Comparing A9000 Hyperscale Manager GUI with IBM Spectrum Control . . . . . . 51
2.5 IBM Spectrum Control reporting options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
2.5.1 Web GUI reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
2.5.2 Create Performance Support Package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
2.5.3 REST API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
2.5.4 IBM Cognos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
2.5.5 Native SQL reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
2.5.6 TPCTOOL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Chapter 3. Setting thresholds and alerts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

3.1 Defining alerts and thresholds in IBM Spectrum Control. . . . . . . . . . . . . . . . . . . . . . . . 72
3.1.1 Alerts and thresholds at device level. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.1.2 Alerts and thresholds for applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.1.3 Alerts and thresholds for general groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
3.1.4 Considerations about using applications and general groups. . . . . . . . . . . . . . . . 82
3.2 Alert and threshold notifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
3.2.1 Event notification in Windows log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
3.2.2 Event notification to be sent by email . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
3.2.3 SNMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

© Copyright IBM Corp. 2018. All rights reserved. iii

 3.2.4 Netcool/OMNIbus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
3.2.5 Trigger a script based on an alert . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
3.2.6 Alert suppressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
3.3 Guidelines for thresholds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
3.3.1 Service level agreements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
3.3.2 Planning and sizings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
3.3.3 Work with your own historical data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
3.3.4 Device-specific metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

Appendix A. Using SSH key for device configuration . . . . . . . . . . . . . . . . . . . . . . . . . 103

Configuring a device using an SSH key . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

iv Managing FlashSystem Performance with IBM Spectrum Control

Preface

This IBM® Redpaper™ publication discusses on performance monitoring for IBM

FlashSystem® storage products. The products reviewed are the IBM FlashSystem FS900,
the IBM FlashSystem V9000, and the IBM FlashSystem A9000 and A9000R.

For each of the FlashSystem devices, the paper reviews performance monitoring options. The
first option is to use features available with the storage management software specific to the
respective devices. The other option, which is the focus of this paper, is to use the IBM
Spectrum™ Control solution.

Using IBM Spectrum Control™ offers the advantage of having a common tool and unique
interface to monitor most devices in your storage infrastructure. This paper explains how to
take advantage of the many monitoring features and reporting options offered by IBM
Spectrum Control. The paper also gives some guidance on how to set appropriate monitoring
thresholds and alerts according to your environment.

Authors
This paper was produced by a team of specialists from around the world working at the
International Technical Support Organization, San Jose Center.

Marion Hejny is an IT Specialist working for the IBM Systems Lab Services team. She works
at the European Storage Competence Center (ESCC), Germany. She provides services for
IBM Spectrum Control including installation, upgrading, and configuration, as well as creating
customized reports for customers in Europe, the Middle East, and Africa. She participates in
the IBM Spectrum Control Beta Program as well as in the Early Adoption Program (EAP)
calls.

Falk Schneider is an IT Specialist working for the IBM Systems Lab Services team. He
works at the European Storage Competence Center (ESCC), Germany. Falk has over
15 years of experience in IT, worked in different roles like IBM Systems Lab services
consultant, ATS, and being responsible for Proof of Concepts (PoC) at the IBM lab in Mainz.
He has also year of field experience, supporting clients with storage and SAN products. His
areas of expertise include disaster recovery (DR), IBM Storage virtualization, and
IBM FlashSystem.

Thanks to the following people for their contributions to this project:

Bert Dufrasne
Stefan Jacquet
Markus Standau
Kevin Webster
IBM

© Copyright IBM Corp. 2018. All rights reserved. v

Now you can become a published author, too!
Here’s an opportunity to spotlight your skills, grow your career, and become a published
author—all at the same time! Join an ITSO residency project and help write a book in your
area of expertise, while honing your experience using leading-edge technologies. Your efforts
will help to increase product acceptance and customer satisfaction, as you expand your
network of technical contacts and relationships. Residencies run from two to six weeks in
length, and you can participate either in person or as a remote resident working from your
home base.

Find out more about the residency program, browse the residency index, and apply online at:
ibm.com/redbooks/residencies.html

Comments welcome
Your comments are important to us!

We want our papers to be as helpful as possible. Send us your comments about this paper or
other IBM Redbooks® publications in one of the following ways:
򐂰 Use the online Contact us review Redbooks form found at:
ibm.com/redbooks
򐂰 Send your comments in an email to:
[email protected]
򐂰 Mail your comments to:
IBM Corporation, International Technical Support Organization
Dept. HYTD Mail Station P099
2455 South Road
Poughkeepsie, NY 12601-5400

Stay connected to IBM Redbooks

򐂰 Find us on Facebook:
http://www.facebook.com/IBMRedbooks
򐂰 Follow us on Twitter:
http://twitter.com/ibmredbooks
򐂰 Look for us on LinkedIn:
http://www.linkedin.com/groups?home=&gid=2130806
򐂰 Explore new Redbooks publications, residencies, and workshops with the IBM Redbooks
weekly newsletter:
https://www.redbooks.ibm.com/Redbooks.nsf/subscribe?OpenForm
򐂰 Stay current on recent Redbooks publications with RSS Feeds:
http://www.redbooks.ibm.com/rss.html

vi Managing FlashSystem Performance with IBM Spectrum Control

Notices

This information was developed for products and services offered in the US. This material might be available
from IBM in other languages. However, you may be required to own a copy of the product or product version in
that language in order to access it.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information on the products and services currently available in your area. Any
reference to an IBM product, program, or service is not intended to state or imply that only that IBM product,
program, or service may be used. Any functionally equivalent product, program, or service that does not
infringe any IBM intellectual property right may be used instead. However, it is the user’s responsibility to
evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The
furnishing of this document does not grant you any license to these patents. You can send license inquiries, in
writing, to:
IBM Director of Licensing, IBM Corporation, North Castle Drive, MD-NC119, Armonk, NY 10504-1785, US

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION “AS IS”

WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A
PARTICULAR PURPOSE. Some jurisdictions do not allow disclaimer of express or implied warranties in
certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may make
improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time
without notice.

Any references in this information to non-IBM websites are provided for convenience only and do not in any
manner serve as an endorsement of those websites. The materials at those websites are not part of the
materials for this IBM product and use of those websites is at your own risk.

IBM may use or distribute any of the information you provide in any way it believes appropriate without
incurring any obligation to you.

The performance data and client examples cited are presented for illustrative purposes only. Actual
performance results may vary depending on specific configurations and operating conditions.

Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm the
accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.

Statements regarding IBM’s future direction or intent are subject to change or withdrawal without notice, and
represent goals and objectives only.

This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to actual people or business enterprises is entirely
coincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, which illustrate programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the sample
programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore,
cannot guarantee or imply reliability, serviceability, or function of these programs. The sample programs are
provided “AS IS”, without warranty of any kind. IBM shall not be liable for any damages arising out of your use
of the sample programs.

© Copyright IBM Corp. 2018. All rights reserved. vii

Trademarks
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines
Corporation, registered in many jurisdictions worldwide. Other product and service names might be
trademarks of IBM or other companies. A current list of IBM trademarks is available on the web at “Copyright
and trademark information” at http://www.ibm.com/legal/copytrade.shtml

The following terms are trademarks or registered trademarks of International Business Machines Corporation,
and might also be trademarks or registered trademarks in other countries.
Cognos® IBM Spectrum™ Redbooks (logo) ®
DB2® IBM Spectrum Control™ Storwize®
HyperSwap® IBM Spectrum Scale™ Tivoli®
IBM® MicroLatency® Variable Stripe RAID™
IBM Cloud™ Real-time Compression™ XIV®
IBM FlashCore® Redbooks®
IBM FlashSystem® Redpaper™

The following terms are trademarks of other companies:

Microsoft, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States,
other countries, or both.

Java, and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its
affiliates.

Other company, product, or service names may be trademarks or service marks of others.

viii Managing FlashSystem Performance with IBM Spectrum Control

 1

Chapter 1. Introduction
This chapter provides a short introduction to IBM FlashSystem products and IBM Spectrum
Control.

This chapter includes the following sections:

򐂰 IBM FlashSystem products
򐂰 IBM Spectrum Control

© Copyright IBM Corp. 2018. All rights reserved. 1

1.1 IBM FlashSystem products
With FlashSystem products, IBM offers a broad range of industry-leading storage
virtualization and data management features to help you address various business needs or
application requirements. IBM FlashSystem devices employ the IBM FlashCore® Technology
implemented in the IBM MicroLatency® modules.
򐂰 For application acceleration, FlashSystem FS900 delivers storage for extreme
performance, enterprise reliability, and operation efficiency.
The FS900 Model AE3 offers performances of up to 1,100,000 I/O per second (IOPS),
bandwidth of up to 10 GBps, and latency as low as 90 microseconds. In addition to the
extreme performance, the unit is efficient and lowers operating costs and increases
efficiency of IT infrastructure by using much less power and space compared to traditional
hard disk drive (HDD) and solid-state disk (SSD) solutions. Another key feature is the
enterprise-class reliability to protect the data with two-dimensional RAID protection, IBM
Variable Stripe RAID™, redundant and hot-swappable components, and concurrent code
loads.
For more information about the latest Flash 900 AE3, see Implementing IBM FlashSystem
900 Model AE3, SG24-8414.
For storage virtualization, FlashSystem V9000 as well as the V5030F and V7000F offer
great flash performance based on the Spectrum Virtualize Code with all its features like
compression, mirroring, and many more functions.
IBM FlashSystem V9000 is a fully integrated Tier 1 all-flash enterprise storage solution
that delivers the full capabilities of IBM FlashCore technology. Its software-defined storage
features simplify data management, improve data security, and preserve your investments
in storage. In addition, it provides a rich set of software-defined storage features, including
IBM Real-time Compression™, dynamic tiering, thin provisioning, snapshots, cloning,
replication, data copy services, and IBM HyperSwap® for high availability.
IBM FlashSystem V9000 improves business application availability. It delivers greater
resource utilization so you can get the most from your storage resources, and achieve a
simpler, more scalable, and cost-efficient IT Infrastructure.
For more information about the functions and how to implement a V9000, see
Implementing IBM FlashSystem V9000, SG24-8413.
򐂰 For the fast growing Hybrid Cloud environments, FlashSystem A9000 and A9000R deliver
high performance-oriented all flash storage with multitenancy.
FlashSystem A9000 and FlashSystem A9000R are functionally built with IBM Spectrum
Accelerate software that is optimized to run on a flash storage grid architecture. Some, but
not all, notable features are the always-on data reduction feature, pattern recognition, and
compression. A9000 and FlashSystem A9000R are designed to deliver high performance,
low latency, and ease of use.
FlashSystem A9000 and FlashSystem A9000R bring together the world-class ease of use
from Spectrum Accelerate software and the microsecond response times that are
provided by IBM FlashCore technology. Designed for enterprise cloud environments, both
FlashSystem A9000 and FlashSystemA9000R offer fast and reliable data storage across
a wide variety of workloads. Both FlashSystem A9000 and FlashSystem A9000R use the
same firmware, and both offer onsite setup and service that are provided by IBM. They
also share a feature set, but significant hardware differences exist. The focus in this paper
is monitoring.
For more information, see IBM FlashSystem A9000 and IBM FlashSystem A9000R
Architecture and Implementation, SG24-8345.

2 Managing FlashSystem Performance with IBM Spectrum Control

1.2 IBM Spectrum Control
IBM Spectrum Control is designed to reduce the complexity of managing SAN storage
devices by allowing administrators to configure and monitor storage devices and switches
from a single console.

Storage admins are continuously faced with the extraordinary challenge of managing very
complex storage infrastructures. Spectrum Control is the product that helps to do just that.
Features like performance monitoring, provisioning, charge-back reporting, and storage
optimization, just to name a few, are all part of this product.

IBM Spectrum Control presents the administrator with a single pane of glass for managing an
entire infrastructure for block, file, and object storage. It provides a complete end-to-end view
of the environment including the possibility to monitor the performance of various storage
entities. It also offers the option to flexibly configure a wide range of customized alerts and
thresholds to inform the administrator if anything is violated.

Storage provisioning is a time consuming task for administrators. System administrators often
must set up new storage for various applications of various levels of criticality. With Spectrum
Control, storage consumers only need to specify the size and required service class for the
new volume or file share. Spectrum Control then determine the “best fit” for the storage
resource that will host the new volume or share.

Beside storage resource management, charge-back reporting is a feature that has gained
more attention in recent releases. It provides easy-to-read reports about storage and a
consumer report that can be emailed to a specific owner of applications, departments,
physical servers, or hypervisors.

Starting with version 5.2.12, Spectrum Control delivers more value for hybrid cloud
environments by using IBM transparent cloud tiering available in IBM Spectrum Scale™. Not
only can you identify file systems in IBM Spectrum Scale clusters that use external storage,
you can also view the capacity of external storage that is being used. This view provides
administrators with an overview of how much storage has been migrated to the cloud. This
information allows the administrators to estimate the possibility of a shortfall if data needs to
be recalled. Alerting was also enhanced to send notifications on the Cloud Gateway Status
during an outage on the external storage.

Further enhancements for IBM Cloud™ Object Storage allow you to monitor the capacity and
space usage of sites and identify which Cloud Object Storage slicestor and accessor nodes
are at each site. Administrators can also see internal resources like sites, mirrors, and vault
space quotas so that they can determine whether any quota limits are being violated.

IBM FlashSystem A9000 and A9000R are now also part of the Spectrum Control support
matrix. Users can view capacity information, space usage, and performance data for the
storage systems. A key part of this family of systems is the data reduction technology that is
integrated in the product. Spectrum Control can report on the data reduction percentage
savings.

For a full list of the features that are provided in each of the IBM Spectrum components, see
the IBM Spectrum Control website.

Chapter 1. Introduction 3
4 Managing FlashSystem Performance with IBM Spectrum Control
 2

Chapter 2. Viewing performance information

This chapter discusses the following topics:
򐂰 Prerequisites for monitoring FlashSystem devices
򐂰 Monitoring performance for FlashSystem 840/ FlashSystem 900
򐂰 Monitoring Performance for FlashSystem V9000
򐂰 Monitoring performance for FlashSystem A9000 or A9000R
򐂰 IBM Spectrum Control reporting options

© Copyright IBM Corp. 2018. All rights reserved. 5

2.1 Prerequisites for monitoring FlashSystem devices
Although setting up a Network Time Protocol (NTP) server and configuring all devices in the
data center to have the time synchronized is relevant for all device monitoring, it is especially
important and necessary when monitoring FlashSystem 840 or FlashSystem 900 devices
with IBM Spectrum Control.

For most of the devices, IBM Spectrum Control uses the time stamp reported by the devices
when saving the performance data in its database. For FlashSystem 840 and
FlashSystem 900, performance data statistics are collected through SNMP. Because SNMP
does not provide the date and time information of the device, the IBM Spectrum Control
server’s time stamp is used.

To use the end to end view provided by IBM Spectrum Control to troubleshoot
performance-related issues across your data center, it is essential to have all your devices
including the IBM Spectrum Control server configured with an NTP server.

Reference: For more information about setting up an NTP server for your devices, see the
Usage of an NTP Server in Storage Area Network environments white paper.

2.1.1 Adding FlashSystem devices to IBM Spectrum Control

Any device that you want to monitor with Spectrum Control must be added to the Spectrum
Control database.

To add a FlashSystem device, complete these steps:

1. Click Add Storage System from the Block Storage Overview in the Spectrum Control
main window, and select FlashSystem Family, as shown in Figure 2-1.

Figure 2-1 Add Storage System window

6 Managing FlashSystem Performance with IBM Spectrum Control

2. After selecting FlashSystem Family, select the appropriate radio button for the
corresponding FlashSystem device that you want to add, as shown in Figure 2-2.

Figure 2-2 Select Flash System model window

Instead of using user name and password for authentication, an SSH key can be used. For
more details, see Appendix A, “Using SSH key for device configuration” on page 103.
3. After devices are added to IBM Spectrum Control, the probe (daily collection of status,
health, configuration, and capacity information (1)) and the performance monitoring (2)
can be scheduled, as shown in Figure 2-3.

Figure 2-3 Schedule probe and Performance monitor

Chapter 2. Viewing performance information 7

 4. If you select the automatic probe schedule as shown in Figure 2-3 on page 7,
IBM Spectrum Control automatically balances all device probes that are set to automatic
probe schedule across the time frame configured in Settings/Automated Probe
Schedule. See Figure 2-4.

Figure 2-4 Automated Probe Schedule

5. When you select manual probe schedule, you can set the time of the day for when the
probe runs on that device. However, it is desirable to have the probes of the individual
devices balanced as much as possible, as shown in Figure 2-5.

Figure 2-5 Manual probe schedule

8 Managing FlashSystem Performance with IBM Spectrum Control

 6. The default monitoring interval is set to every minute. The default can be changed to a
longer interval, as shown in Figure 2-6.

Figure 2-6 Performance monitor interval

One-minute performance data is kept for seven days, but they are also aggregated to
five-minute samples, and hourly and daily data. The retention time settings for samples, and
hourly and daily data can be configured in the History Retention time settings window in
IBM Spectrum Control, as shown in Figure 2-7.

Figure 2-7 History Retention time settings

Chapter 2. Viewing performance information 9

 Because all performance monitoring data, like any other data that IBM Spectrum Control
collects are kept in a DB2® database called TPCDB, a compromise between data accuracy,
retention time settings, database size, CPU, and memory of the Spectrum Control server is
necessary.

When collecting performance data, IBM Spectrum Control collects the device’s performance
counters after each interval. After two intervals, the delta is calculated and divided by the
interval. Therefore, the shorter the interval, the more likely peaks will be visible.

2.1.2 Considerations for Performance Monitoring with IBM Spectrum Control

After performance data is collected, it can be viewed in the IBM Spectrum Control GUI, at
different levels, for the complete Storage System and for almost all of its internal resources.

Accessing performance data in IBM Spectrum Control is consistent across all levels. The
examples in this chapter are for A9000 volumes, as shown in Figure 2-8.

You can access the performance charts using these methods:

򐂰 Selecting the Performance tab (1) opens the performance chart to display the top five
volumes with the most total I/O. Depending on the number of volumes in Spectrum
Control, this operation can take a few seconds.
򐂰 Using sorting and filtering, you can identify and highlight the volumes of interest. Click
View Performance (2) to open a new window with the performance chart.

1 2

Figure 2-8 View performance.

10 Managing FlashSystem Performance with IBM Spectrum Control

 in Figure 2-9, the performance chart shows the performance graphs for the individual
volumes (1) and the selected metrics (2). The time frame can be either relative or absolute.
The performance graphs can be opened in a new window (5). The URL of the chart being
displayed can be bookmarked for future use. The metrics of the table underneath the chart
can be selected by clicking Edit Table Metrics (6) and can be sorted (7). The selected and
highlighted volumes are then shown in the chart.

3
1 5

4
6 7

Figure 2-9 Performance chart tangerine

The metrics to be displayed can be chosen as shown in Figure 2-10

Figure 2-10 Select Performance metrics

Up to 10 different volumes can be selected. Up to six different metrics with two different units
can be displayed at the same time because only two y-axis can be displayed.

Although displaying each individual volume and displaying each individual metric can be
toggled on and off, it might be confusing having that many graphs in one chart.

Chapter 2. Viewing performance information 11

 Figure 2-11 shows the performance for only one volume, although three were initially
selected, because two of them were deselected (1). The same behavior applies to the
performance metrics where only two of the six metrics were selected to be displayed (2).

Figure 2-11 Volume performance chart in IBM Spectrum Control

To show more metrics with different units, the performance data can also be displayed as a
table by using the toggle chart/table function (1) and then adding additional metrics as
columns to the table (2), as shown in Figure 2-12.

Figure 2-12 Table view of performance chart

Depending on the time frame that is selected (and the history retention time settings), the
granularity of the performance chart changes automatically:
򐂰 One minute sample data, if collected, is shown for one hour, six hours, 12 hours, or one
day range. Otherwise, the five minute sample data is used.
򐂰 Five minutes sample data is shown for one week.
򐂰 Hourly data is shown for one month.
򐂰 Daily data is shown for one year.

12 Managing FlashSystem Performance with IBM Spectrum Control

 When you decrease the granularity, the data accuracy also decreases. For the time range of
one day, the one-minute sample data is available, as well as aggregated hourly data. The
granularity can be manually adjusted. Although the averages of Total I/O and Response Time
are almost the same, the maximum values differ a lot, which can be seen in the table
underneath the graphs but can already be assumed when looking at the graphs.

Comparison of sample versus hourly data can be seen in Figure 2-13 and in Figure 2-14.

Figure 2-13 Sample data

Figure 2-14 shows the hourly data for the same period.

Figure 2-14 Hourly data

For troubleshooting purposes, the displayed time frame should be small enough so that the
smallest interval data can be shown.

Chapter 2. Viewing performance information 13

 Figure 2-15 illustrates the drill up and drill down capability available in IBM Spectrum Control.

Because in this example the V9000 write response time shows a peak at a certain time, this
functionality can be used to show for the same time frame the V9000’s pools or V9000’s
volume write response time to determine the root cause of those peaks.

Figure 2-15 Drill up and drill down functionality

The new performance charts open in separate windows and can be arranged for comparison
purposes.

14 Managing FlashSystem Performance with IBM Spectrum Control

 The date and time settings of multiple windows can be synchronized by using the
Synchronize Time feature, as shown in Figure 2-16.

Figure 2-16 Synchronize time

2.1.3 Special considerations for FlashSystem devices

At the time of writing, IBM Spectrum Control version (v 5.2.15) is limited to showing Fibre
Channel ports only. If the FlashSystem device is equipped with InfiniBand, iSCSI, or FCoE
ports, these ports will not be displayed in IBM Spectrum Control. In this case, no port
performance data will be collected.

Performance metrics that are only available in the native FlashSystem GUI can be accessed
easily in IBM Spectrum Control by clicking Actions → Open Storage System GUI, as shown
in Figure 2-17 on page 16.

Chapter 2. Viewing performance information 15

Figure 2-17 Open Storage System GUI functionality in IBM Spectrum Control

Some FlashSystem devices like the V9000 also offer a direct link to the system’s performance
graphs by clicking Action → View Real-Time Performance, as shown in Figure 2-18. This
function can also be used to view real-time performance values if the one-minute interval data
that can be shown in IBM Spectrum Control is not detailed enough.

Figure 2-18 View real-time performance

16 Managing FlashSystem Performance with IBM Spectrum Control

2.2 Monitoring performance for FlashSystem 840/
FlashSystem 900
This section discusses performance monitoring for FlashSystem 840 and FlashSystem 900,
first using the system Storage Management native tools and then using Spectrum Control.

2.2.1 Monitoring FlashSystem 840 an FlashSystem 900 using native tools

You can monitor the performance by using the system native tools available from the
FlashSystem built-in Storage Management GUI. This GUI is web-based and can be used with
any browser pointed at the management IP address of the FlashSystem 840 or 900 to be
monitored. Note that Java Script and cookies must be enabled in your browser.

Log in to the GUI

To log in and open the Monitoring Performance window, complete these steps:
1. Enter your login credentials in the login window, as shown in Figure 2-19. Because the
performance monitor is just a monitoring tool, any user role can be used here, and
specifically, the monitor role.

Figure 2-19 Log-in window for a Flash 840

2. After a successful login, you can open the Monitoring function by clicking the Monitoring
icon, and then selecting Performance, as shown in Figure 2-20.

Figure 2-20 Opening the Monitoring Performance window

Chapter 2. Viewing performance information 17

 Performance Monitor
Figure 2-21 shows the Performance Monitor window. Labels were added to indicate specific
areas of interest.

Graphs menu Shown values

Time slider

Selected Predefined time buttons

Performance summary timeframe

Figure 2-21 Performance Monitor Entry window

By default, the display shows the average latency for read and write operations at a system
level. The x-axis shows the time frame (default is 10 minutes) and the y-axis shows selected
value, such as latency in this example.

The time frame can be changed by clicking the Time buttons:

򐂰 now
򐂰 1 hour
򐂰 1 day
򐂰 1 week
򐂰 all

The “all” selection includes all the collected data, for up to 300 days.

At the bottom of the diagram, a slider can be used to display the targeted time frame for
further analysis of this period. This slider has two parts:
򐂰 One part to set the starting point
򐂰 One part to set the endpoint to focus on a shorter time frame or one in the past.

The selected start and end time or date are also shown under the x-axis.

18 Managing FlashSystem Performance with IBM Spectrum Control

Graphs menu
The Graphs menu is accessed from the upper left of the Performance Monitoring window.
The following different types of graph can be selected:
򐂰 System I/O
򐂰 System Latency
򐂰 System Bandwidth
򐂰 Interface Port Total IOPS
򐂰 Interface Port Total Queue Depth

Figure 2-22 shows the Graphs menu with five predefined graphs. Other graphs can be
created as described in “Customize chart tab” on page 20. The different graphs can be
chosen based on specific needs. This figure shows the menu for Fibre Channel ports. If there
are other ports installed in the storage system, such as iSCSI or InfiniBand, then they are
shown as well.

Figure 2-22 Graphs menu

The content and meaning of the selectable graphs is as follows:

򐂰 System I/O
The System I/O graph displays the average number of read, write, and total I/O operations
per second (IOPS) over the sample period. Each request type (read, write, and total) is
represented by a different color line.
򐂰 System Latency
The System Latency graph displays the average amount of time in milliseconds (ms) that
each read and write I/O request takes over the specified sampling period. Each request
type (read and write) is represented by a different color line. The user can see the average
or the maximum latency, but not both at the same time.
򐂰 System Bandwidth
The System Bandwidth graph displays the average number of megabytes per second
(MBps) of read, write, total, and rebuild bandwidth over the sample period. Each
bandwidth type (read, write, total, and rebuild) is represented by a different color line.
There is one line graph for each system that is selected.
򐂰 Average Port IOPS
The Average Port IOPS graph displays the average number of read, write, and total IOPS
for all ports on all adapters over the sample period.
򐂰 Average Port Queue Depth
The Average Port Queue Depth graph displays the average number of operations of that
type for all ports on all adapters over the sample period.

Chapter 2. Viewing performance information 19

 Customize chart tab
The default graphs can be edited and customized. Click below the graphs menu icon on the
left side to select the Resources. Figure 2-23 shows some options to customize the graphs at
a system level.

resources to be customized

included resources
customize chart tab

available metrics for this resource

Click when finished

Figure 2-23 Customize graphs

The resources to be customized can be selected in the upper right corner of this menu. The
following available resources are available for customization:
򐂰 System
򐂰 Flash Modules
򐂰 Interface ports

Based on the selected level, different set of resources can be selected. At the system level,
configured systems can be selected. In our example, there is only one system. At the Flash
module level, up to six modules can be selected. At the interface port level, up to six ports can
be selected. If more than six are chosen, they are summarized as a single resource with a
single line in the chart.

If the selected resource is an interface, the representation is particular: For each port, a
corresponding line is shown, but the color is based on the adapter. So if three ports are
selected, one from adapter 1 and two from adapter 2, three lines are shown in the graph (one
for each adapter and port), but only two colors are shown (one color per adapter). This
system can be confusing at first.

20 Managing FlashSystem Performance with IBM Spectrum Control

 Figure 2-24 shows this situation in detail. In the upper right corner is the caption that lists only
Host Adapter 1 Canister 1 and Host Adapter 2 Canister, each with its own color. Because
three ports are selected in the customized chart, they are all shown in the chart. As visible in
the detail box, there are three lines shown with the color based on the HBA.

Caption

Exact values at this point in time

Data for every single port

Figure 2-24 Chart for Fibre Channel ports

By clicking the graph at a particular point in time, all values for the existing lines are shown in
a single box. By using this function, exact values can be made visible.

The metrics shown in the chart depend on the selected resource. Not all metrics can be
shown at the same time. Only compatible metrics can be shown within the same chart. See
also Figure 2-42 on page 34. There is no option for a second y-axis.

For a complete table of the available metrics, see Table 2-1 on page 32 for the native or
built-in area. For a detailed description of every metric, select Help Contents, as shown in
Figure 2-25.

Figure 2-25 Performance Help

Chapter 2. Viewing performance information 21

 Equipped with all this information, users can now build their own customized graphs to
monitor different metrics. If you want to compare different metrics that cannot be shown in a
single chart, you can easily add a second chart to the view. For instance, if you want to
compare IOPS with latencies, bring them into the same view by clicking the symbol shown in
Figure 2-26.

Figure 2-26 Adding a second chart

In the customized chart, the resources and metrics can be selected. An illustration is shown in
Figure 2-27.

Figure 2-27 View with two charts

22 Managing FlashSystem Performance with IBM Spectrum Control

Both graphs have the same scale in the x-axis because they are by default linked. To view
different time frames or have a different scale on the x-axis, the charts can be unlinked by
clicking the small symbol in the middle on the right side, as shown in Figure 2-28. You can
click the same symbol link the charts again.

Figure 2-28 Unlink charts

You can create a bookmark to customized charts that are important or used often. This
bookmark adds those charts to the Favorites in the Charts menu. A chart that is often used or
shared by a team, it can be added to the predefined charts for easy access and sharing. To
add a chart to the Charts menu, create your chart and then click the small yellow star on the
right side of the chart. You also must give it a name. When a user adds a chart as a favorite,
as shown in Figure 2-29, it becomes available to every other user.

Figure 2-29 Adding and naming the chart

The new favorite chart becomes accessible by using the Charts menu. Here the favorite chart
can easily be selected, renamed, and deleted by selecting the appropriate action from the
pull-down menu, as shown in Figure 2-30.

Figure 2-30 Chart menu with a new favorite

Chapter 2. Viewing performance information 23

 From the same menu, you can also set your favorite chart (or any predefined chart) as the
default chart or pin it to the toolbar. When added to the toolbar, the chart can be opened with
a single click from the toolbar. Figure 2-31 shows how the icons are placed.

Figure 2-31 Icon placement

When the mouse cursor is placed over the icon, it is magnified and the name of that chart is
shown.

Finally, to download and export the values from the chart, click the Disk icon near the
Favorites icon. As a result, a comma-separated values (csv) file is created with all the
metrics included in the chart for the complete time frame.

A tool is available on IBM Techdocs that offers an extra way to obtain performance data in an
csv file format that includes more metrics. It is called the IBM Flash System Performance
Statistics Data Capture and Extraction Procedure.

With this procedure, all the data that is collected by the GUI can be downloaded and
formatted as a csv file. The file can be used for further analysis in performance studies or in a
proof of concept scenario. Just download the files and follow the documentation.

2.2.2 Monitoring FlashSystem 840 / 900 with IBM Spectrum Control

To monitor performance for the s FlashSystem 840 or FlashSystem 900 with IBM Spectrum
Control, enable the SNMP agent on the FlashSystem device to allow Spectrum Control to
collect performance statistics. The SNMP agent on the FlashSystem device is disabled by
default.

Note: FlashSystem 840 and FlashSystem 900 only support SNMP v1, which does not
have encryption capability.

24 Managing FlashSystem Performance with IBM Spectrum Control

SNMP service on the FlashSystem 840/900 can be easily turned on from the Management
GUI, as illustrated in Figure 2-32.

Figure 2-32 FlashSystem 900 Web UI

Alternatively, you can enable SNMP from the FlashSystem CLI as shown in Example 2-1.

Example 2-1 Enabling SNMP from CLI

IBM_FlashSystem:FlashSystem-840-03:superuser>lssnmpagent
enabled no
community ibmpublic
IBM_FlashSystem:FlashSystem-840-03:superuser>chsnmpagent -enable yes
IBM_FlashSystem:FlashSystem-840-03:superuser>lssnmpagent
enabled yes
community ibmpublic
IBM_FlashSystem:FlashSystem-840-03:superuser>

Chapter 2. Viewing performance information 25

 If you try to schedule performance monitoring without having the SNMP agent enabled on
FlashSystem 840 or FlashSystem 900, you are prompted to enable it. Figure 2-33 shows the
Data Collection Schedule window.

1
2

Figure 2-33 Data Collection Schedule window

The SNMP community is automatically detected during the probe. If it is necessary to change
the community, then the FlashSystem 840 or FlashSystem 900 might need to first be probed
again to continue with the performance data collection.

Important: If your device is behind a firewall, port 161 must be open to get performance
data. For more information about ports that need to be open, see the IBM Spectrum
Control section of IBM Knowledge Center.

You can use the Hyperlink (1) to open the FlashSystem GUI to enable SNMP as shown in
Figure 2-33.

Clicking Test SNMP (2) checks whether SNMP is turned on by using the read community to
get a confirmation.

26 Managing FlashSystem Performance with IBM Spectrum Control

Either method opens the default Data Collection window, as shown in Figure 2-34.

Figure 2-34 Data Collection Schedule

Performance Monitoring for the FlashSystem can now be scheduled.

Because no individual volume performance data is available yet, the Overview window
(Figure 2-35 on page 28) shows the following charts:
򐂰 Overall System Activity
– I/ORate (Read/Write/Total)
– Data Rate (Fed/Write/Total)
– Response Time (Read/Write/Total) 1
򐂰 Most Active Nodes
– I/O Rate (Total - per node)
– Data Rate (Total - per node)
򐂰 Drive Activity
– Back-end Data Rate (Total - per drive)
– Flash Health Percentage (per drive)
򐂰 Most Active Ports
– Bandwidth (Overall - per Port)

1 If FlashSystem 840/FlashSystem 900 Firmware version is 1.4.5 or later.

Chapter 2. Viewing performance information 27

Figure 2-35 FlashSystem 840 / FlashSystem 900 Overview window

Reporting options are described in 2.5, “IBM Spectrum Control reporting options” on page 55.

2.2.3 Performance metrics for IBM FlashSystem 840 / 900

To get performance metrics, the FlashSystem 840 or FlashSystem 900 needs to be at
firmware version 1.4.4. or later. To get response time values, the firmware version needs to be
1.4.5 or later.

The FlashSystem 840 or FlashSystem 900 does not provide any performance statistics for
volumes. Therefore, there is also no performance data for pools and RAID arrays.

Available performance statistics are primarily for ports. Node and overall FlashSystem 840 or
FlashSystem 900 performance statistics are calculated or aggregated from port performance
data.

For FlashSystem 840 or FlashSystem 900 that have no Fibre Channel adapters, performance
data is only available for the drives.

28 Managing FlashSystem Performance with IBM Spectrum Control

 For the FlashSystem 840 and FlashSystem 900, the Performance tab (1)/ the View
Performance Button (2) and the View Performance action (3) are enabled for ports, drives,
nodes and for the overall FlashSystem 840 or FlashSystem 900 system performance.
Figure 2-36 shows how to select port performance for FlashSystem 840 / 900.

Figure 2-36 Selecting Port Performance for FlashSystem 840 / FlashSystem 900

Chapter 2. Viewing performance information 29

 Available Port Performance metrics
Figure 2-37 shows the selection window for Available Port Performance metrics for
FlashSystem 840 and FlashSystem 900.

Figure 2-37 Port performance metrics for FlashSystem 840/FlashSystem 900

Available Flash module performance metrics

For the Flash modules, which are shown as drives in IBM Spectrum Control, a metric called
Flash Health Percentage is available. That percentage decreases when your flash modules
suffer degradation. The Flash Health Percentage is calculated based on the number of
unusable blocks in the flash module.

Because access to the drives is not I/O based but more like a type of Direct Memory Access,
no Back-end I/O Rate metrics are available for the flash modules. Only Data Rates can be
reported.

IBM Spectrum Control supports a range of different storage systems that use different flash
devices embedded in different kind of modules, such as MicroLatency modules. The generic
term disks is used instead of MicroLatency modules.

Figure 2-38 shows the available Performance Metrics for the FlashSystem 840 / 900 Flash
modules.

Figure 2-38 Flash Module Performance metrics for FlashSystem 840 / FlashSystem 900

Although the performance chart implies that the FlashSystem 840/FlashSystem 900 can
report back-end response time for drives, this metric is not available.

30 Managing FlashSystem Performance with IBM Spectrum Control

 Available Node Performance metrics
The Nodes’s Volume metrics are calculated from the Port metrics and show the same values
as the Port metrics except the Transfer Size, which is not available in the Volume Metrics tab.

Figure 2-39 shows the selection window for Available Node Performance Metrics for the
FlashSystem 840/FlashSystem 900.

Figure 2-39 Node Performance Metrics for the FlashSystem 840/FlashSystem 900

Although the performance chart implies that the FlashSystem 840/FlashSystem 900 can
report on Node’s System CPU Utilization, this metric is not available.

Available overall FlashSystem 840 / 900 System Performance Metrics

Figure 2-40 shows the selection window for the overall FlashSystem 840/FlashSystem 900
Port Performance metrics.

Figure 2-40 Overall FlashSystem 840 / FlashSystem 900 Port Performance metrics

Chapter 2. Viewing performance information 31

 Figure 2-41 shows the window for Overall FlashSystem 840/FlashSystem 900 Disk
Performance metrics.

Figure 2-41 Overall FlashSystem 840 / FlashSystem 900 Disk Performance metrics

The Volume’s Metrics tab shows the sum of the measured performance of all the ports in the
storage system, and therefore shows the same values as the Port Metrics tab, except for the
Transfer Size metrics.

2.2.4 Comparing native GUI and Spectrum Control

Table 2-1 shows an overview of available Performance Metrics and where they can be
displayed, in the FlashSystem 840 / FlashSystem 900 GUI or in Spectrum Control.

Table 2-1 FlashSystem 840/900 Performance Metrics

Level Metric Native with IBM
Spectrum
Control

Storage System IO in ops/s: Read/Write/Total X X

Storage System Bandwidth/Data Rate in MiB/s: Read/Write/Total X X

Storage System Total Rebuild Bandwidth in MiB/s X

Storage System Latency/Response Time in ms/op: Read/Write X X

Storage System Overall Latency/Response Time in ms/op X

Storage System Max Latency/Response Time in ms/op: X

Read/Write

Flash Module / Drives Flash Health % X X

Flash Module / Drives Power in Watts X

Flash Module / Drives Garbage Collection: User Data %: Hot/Warm/Cold X

Flash Module / Drives Garbage Collection: User Data %: Total Write X

Amplification Factor

Flash Module / Drives Background Health Checker: Health Checker X

Operations: Iterations/Blocks Checked/Wear
Leveling Moves

Flash Module / Drives Background Health Checker: Total Completion X

Percentage

Flash Module / Drives Data Rate (MiB/s): Read/Write/Total X

32 Managing FlashSystem Performance with IBM Spectrum Control

Level Metric Native with IBM
Spectrum
Control

Port IO in ops/s: Read/Write/Total X X

Port Total Read Modify Writes (Operations) X

Port Bandwidth/Data Rate in MiB/s: Read/Write/Total X X

Port Bandwidth Percentage: Send/Receive/Overall X

Port Queue Depth Operations X

Port Aligned Command Size Operations: Small/Large X

Port Block Size Operations: Small/Large X

Port Response Time in ms/op: Send/Receive/Overall X

Port Transfer Size (KiB/op): Send/Receive/Overall X

Node: Port metrics IO in ops/s: Send/Receive/Overall X

Node: Port metrics Data Rate in MiB/s: Send/Receive/Overall X

Node: Port metrics Response Time in ms/op: Send/Receive/Overall X

Node: Port metrics Transfer Size (KiB/op): Send/Receive/Overall X

Historical data can be seen with native tools for the last 300 days. With IBM Spectrum
Control, historical performance data can be seen as configured in the retention time settings
as shown in Figure 2-7 on page 9.

Chapter 2. Viewing performance information 33

 With the native FlashSystem 840/900 GUI, you can either select multiple resources or
multiple metrics, as shown in Figure 2-42.

Figure 2-42 FlashSystem 840/900 metrics selection window

34 Managing FlashSystem Performance with IBM Spectrum Control

 You can also select multiple resources and multiple metrics in IBM Spectrum Control, as
shown in Figure 2-43.

Figure 2-43 Multiple resources and multiple metrics in IBM Spectrum Control

In the FlashSystem 840/900 native storage management GUI, you can easily open a split
window to show different resources or different metrics. For example, you can show the
systems performance in the upper part of the window and the port performance in the lower
window, as illustrated in Figure 2-44.

Figure 2-44 FlashSystem 840/900 split Performance window

Chapter 2. Viewing performance information 35

 In IBM Spectrum Control, you can open different performance windows with the Open in new
window function and align them to compare the data, as shown in Figure 2-45.

Figure 2-45 Arranging multiple Spectrum Control windows

An advantage of opening and arranging multiple windows with IBM Spectrum Control is that
you can easily compare different time ranges. With IBM Spectrum Control, as many windows
can be opened and arranged as necessary, whereas with the native FlashSystem 840/900
GUI, the performance chart can only be split into two windows.

The performance charts on the FlashSystem 840/900 are updated every five seconds,
whereas in IBM Spectrum Control the minimum interval is one minute. To get real-time
performance data, the native FlashSystem 840/900 GUI can be opened from IBM Spectrum
Control as shown in Figure 2-17 on page 16.

Although the port performance charts in the FlashSystem 840/900 GUI use the same color
for ports that are connected to the same adapter as shown in Figure 2-24 on page 21, IBM
Spectrum Control shows each port with its own color, independently of which adapter it is
connected to, as shown in Figure 2-30 on page 23.

36 Managing FlashSystem Performance with IBM Spectrum Control

2.3 Monitoring Performance for FlashSystem V9000
This section describes performance monitoring for FlashSystem V9000, first using the system
storage management native tools, and then using Spectrum Control.

2.3.1 Monitoring FlashSystem V9000 with the native tools

FlashSystem V9000 can be managed and monitored from the system GUI or from the CLI. To
use the web-based GUI, complete these steps:
1. Enter the system IP address in your browser.
2. Enter your credentials to get access, as shown in Figure 2-46. The examples are based on
Version 7.x.

Figure 2-46 V9000 log-in window

3. From the main widow, select Monitoring → System → Performance to open the
Performance monitor, as shown in Figure 2-47.

Figure 2-47 Menu to enter the Performance monitor

The Performance Monitor window, shown in Figure 2-48 on page 38, is divided into four
sections of equal size that provides information about performance of the different areas.
Their location and purpose is:
– CPU Utilization (upper left). Here the usage is shown for these metrics:
• System CPU %
• Compression CPU % (only when compression is enabled)

Chapter 2. Viewing performance information 37

 – Volumes (upper right). This view shows the overall volume data for these metrics:
• Read - Total read traffic of the system
• Write - Total write traffic for the system
• Read latency - Average read time per volume
• Write latency - Average write time per volume
– Interfaces (lower left). The overall statistic for every available port is shown here divided
into these metrics:
• FC - All Fibre Channel traffic
• iSCSI - All iSCSI traffic
• SAS - All serial attached SCSI traffic, backend, and frontend
• IP Remote Copy - Remote Copy traffic over IP connections
– Mdisks (lower right). The overall statistics for MDisks for these categories:
• Read - Total read traffic of the system
• Write - Total write traffic for the system
• Read latency - Average read time per volume
• Write latency - Average write time per volume

Figure 2-48 V9000 Performance monitor

This view is fixed. All four charts are always displayed at the same place. The monitored time
frame is 5 minutes for every chart in this window, giving you the same time frame view of all
metrics. In the upper left of the window, two pull-down menus allow you to customize the
views. The System Statistics menu can be used to select a single control enclosure (here
called node) or the whole system. If only one control enclosure needs to be monitored, it can
be selected here. The second pull-down menu allows you to display IOPS or MBps, as shown
in Figure 2-49.

Figure 2-49 Changeable settings for the V9000 Performance monitor

38 Managing FlashSystem Performance with IBM Spectrum Control

 For each of the four charts, you can select or clear the measurements to show. The choice
depends on the resources being monitored (volume, MDisk, interfaces, or CPU). Because
data is collected continuously, after you select a metric for display, you can immediately see
the last 5 minutes of collected data. See Figure 2-50.

Figure 2-50 Volume performance

To get a detailed value for a single point in time, you can click the curve in every chart and see
the detailed values along with the time when those items were measured. Figure 2-51 shows
an example for the Interfaces chart.

Figure 2-51 Exact value in Performance Monitor

A general view for the complete system is always visible at the bottom of every window in the
GUI. It displays as a dashboard and shows real-time values, as illustrated in Figure 2-52.

Figure 2-52 Performance summary visible on every GUI window on the V9000

Chapter 2. Viewing performance information 39

 You can also monitor the V9000 performance through the CLI. The CLI interface uses the
same IP address as the GUI. To open a CLI session, complete these steps:
1. Open a Secure Shell (SSH) session.
2. Enter your user ID and password (alternatively, a key can be provided, which is more
convenient for scripts and automation) to authenticate with the CLI session.
3. Use the lsnodestats and lssystemstats commands to get real-time performance data.

Note: From the CLI, the term node is often used instead of control enclosure. In any
output, the term node really stands for control or expansion enclosure.

To get the overall system performance data, issue the lssystemstats command. This
command displays the most recent values of all node statistics in the system. The command
output is a list of all collected statistics for the last available point in time. This output can be
formatted with the standard CLI delimiters and filters. A shortened output is shown in
Example 2-2.

Example 2-2 Output of lssystemstats

IBM_FlashSystem:Cluster:superuser>lssystemstats
stat_name stat_current stat_peak stat_peak_time
compression_cpu_pc 0 0 170811015618
cpu_pc 2 2 170811015618
fc_mb 0 13 170811015353
fc_io 614 691 170811015353
sas_mb 0 0 170811015618
sas_io 0 0 170811015618

To get a history of a single statistic like the CPU usage (cpu_pc) in the above example, the
-history parameter can be used, as shown in Example 2-3.

Example 2-3 Displaying a specific statistic

IBM_FlashSystem:Cluster:superuser>lssystemstats -history fc_mb
sample_time stat_name stat_value
170811015621 fc_mb 320
170811015626 fc_mb 411
170811015631 fc_mb 442
170811015636 fc_mb 404
170811015641 fc_mb 527

The lsnodestats command shows the same statistics, but based on a single node (that is,
enclosure) in the system. That is why two additional columns are in use for the Node ID and
Node name. The other columns are the same as for the complete system. In total, there are
six columns:
node_id ID of the reporting node
node_name Name of the node
stat_name Name of the statistic that is listed
stat_current Current value of the statistic
stat_peak Peak value of the listed statistic during the last 5 minutes
stat_peak_time Timestamp of the peak value

40 Managing FlashSystem Performance with IBM Spectrum Control

The values are shown for each node separately. This command can be used if only a single
enclosure or node is in focus or needs to be checked. The -history parameter can be used
also, but must be used with the node ID or node name as additional parameter. Example 2-4
shows the output of the lsnodestats command.

Example 2-4 lsnodestats command output

IBM_FlashSystem:Cluster_Hotel:superuser>lsnodestats
node_id node_name stat_name stat_current stat_peak stat_peak_time
1 node1 compression_cpu_pc 0 0 170811023200
1 node1 cpu_pc 2 2 170811023200
1 node1 fc_mb 0 0 170811023200
1 node1 fc_io 306 315 170811023100
1 node1 sas_mb 0 0 170811023200
1 node1 sas_io 0 0 170811023200

The performance monitoring capability described here and included with the V9000 native
storage management GUI is for real-time monitoring over a short period. Spectrum Control
offers the advantage of collecting performance data over an extended period (longer history)
and covering more aspects of the system.

The FlashSystem V9000 management GUI collects data for Mdisks, volumes, nodes, and
SAS drives automatically. The data collected is part of a support file called snap, but the data
can be downloaded without downloading the complete snap.

Restriction: The data for Flash modules is not included in the drive data collection.

The data is stored in a single file for every interval. Such an interval can be from 1 minute to
up to 60 minutes. The default value is 5 minutes. The limitation here is that the data is stored
for 16 intervals maximum. When the 17th interval is started, the oldest data set is deleted.
Therefore, the interval time also sets the maximum monitored time frame. If the interval is set
to two minutes, 32 minutes of data is stored in the files. However, the default value of
five minutes brings 80 minutes of performance data, but the granularity is then five minutes.
So, depending on what situation you need to analyze, it is important to decide upfront what
data granularity is required.

The interval time can be set by using the startstats command. To get all the details and
possible parameters, see the command help or consult IBM Knowledge Center.

Keep in mind that the last setting will remain in effect until it is changed again. If the interval
was changed for a specific scenario or maybe just for testing, make sure that it is set back to
your standard interval to avoid unexpected results in the next data collection.

Note: Do not use the startstats command to change the interval if you use an external
monitoring software such as IBM Spectrum Control. Change the interval in the monitoring
software instead.

The data is collected on a per node basis and not for the complete cluster at a single location.
Even though the samples are taken at the same time on all nodes, the data files are stored on
their respective collecting nodes. To get all the files for members in the cluster, the data must
be copied to the config node before download.

Chapter 2. Viewing performance information 41

 The files are stored in the /dumps/iostats directory of each node. To collect the data on the
config node, use the cpdumps command to consolidate before download. The file names have
the following format:
򐂰 MDisk statistics:
Nm_stats_<node_serial_number>_<date>_<time>
򐂰 VDisk statistics:
Nv_stats_<node_serial_number>_<date>_<time>
򐂰 Node statistics:
Nn_stats_<node_serial_number>_<date>_<time>
򐂰 Disk drive statistics (not for IBM Flash System V9000 Flash Modules):
Nd_stats_<node_serial_number>_<date>_<time>

The <node_serial_number> gives the information about which node or control enclosure the
data is collected, <date> is the day of collection in the form <yymmdd>, and the <time> shows
the exact time stamp in the form <hhmmss>.

The filename Nv_stats_78AV320_170810_182416 is an example of a file for volume data,

collected on the Node with the serial 78AV320, on Aug 10th at 06:24pm. To list all collected
data, use the lsdumps command with the prefix iostat, as shown in Example 2-5.

Example 2-5 lsdump command

IBM_FlashSystem:V9000_tpcv9k1:superuser>lsdumps -prefix /dumps/iostats
id filename
0 Nv_stats_78AV320_170810_182416
1 Nm_stats_78AV320_170810_182416
2 Nd_stats_78AV320_170810_182416
3 Nn_stats_78AV320_170810_182416
4 Nd_stats_78AV400_170810_182416
5 Nn_stats_78AV400_170810_182416
6 Nv_stats_78AV400_170810_182416

Tip: To copy the files from the Flashsystem V9000 to a local workstation the pscp
command, included with PuTTY, can be used. Open a command prompt and change to the
directory where pscp.exe is located.
C:\pscp>pscp -unsafe -load performancedata [email protected]:/dumps/iostats/*
c:\v9000data
Using keyboard-interactive authentication.
Password:
Nd_stats_78AV460_170811_0 | 1 kB | 1.4 kB/s | ETA: 00:00:00 | 100%
d_stats_78AV460_170811_0 | 1 kB | 1.4 kB/s | ETA: 00:00:00 | 100%
Nd_stats_78AV460_170811_0 | 1 kB | 1.4 kB/s | ETA: 00:00:00 | 100%

Note the following about this output:

򐂰 The -unsafe parameter is needed because wildcards are in use
򐂰 A PuTTY session with the name behind the -load needs to be configured up front. If
there is no key stored, pscp prompts for a password, as shown in the above example.

A complete list with a short explanation of all collected data can be found in IBM Knowledge
Center.

42 Managing FlashSystem Performance with IBM Spectrum Control

2.3.2 Monitoring V9000 with IBM Spectrum Control
In terms of performance monitoring, IBM Spectrum Control basically captures the same
information for FlashSystem V9000 as for any other IBM Storwize® family system.

A complete list of performance metrics available for V9000 is listed in the IBM Spectrum
Control section of IBM Knowledge Center.

The flash modules can be seen in the disks window, as illustrated in Figure 2-53.

Figure 2-53 V9000 Flash modules in IBM Spectrum Control

Reporting options are described in 2.5, “IBM Spectrum Control reporting options” on page 55.

Drive performance data

The V9000 Nd_stat file contains only statistics of SAS drives, not for Flash modules in a Flash
storage enclosure. Therefore, drive performance data cannot be shown in IBM Spectrum
Control. For more information, see the IBM FlashSystem V9000 section of IBM Knowledge
Center.

Because you can also add enclosures with SAS drives to a V9000, the disks performance tab
is available in IBM Spectrum Control for V9000 disks.

Chapter 2. Viewing performance information 43

 Zero Buffer Credit Percentage/Port Send Delay I/O Percentage
The hardware design of the 16 Gb HBAs has changed and they do longer provide zero buffer
credit counters. In the past (up until version 5.2.13) Spectrum Control showed an incorrect
value of 0.00 for Zero Buffer Credit Timer and Zero Buffer Credit Percentage (%) for 16 Gb
ports (as reported by the Storwize systems). Starting with With IBM Spectrum Control v
5.2.14, N/A (rather than 0.00) is shown for these metrics for 16 Gb ports only.

Because the lack of the Zero Buffer credit counters means that there is no way to debug
slow-draining device problems in the SAN, new counters (dtdt, dtdc, and dtdm) were
introduced with Storwize version 7.8.1. These counters will attempt to measure sent I/O delay
by retrieving a subset of traffic that can be measured by software. Therefore the new counters
do not replace the zero buffer credit counters. Instead, they represent the delay expected in
transmitting data calculated based on the amount of data queued in the port and the
estimated port speed. These new counters are available in IBM Spectrum Control starting
with version 5.2.14. They can be displayed using the Port Send Delay Time (ms/op), Port
Send Delay I/O Percentage (%), or both.

These new counters and therefore the new metrics Port Send Delay Time (ms/op) and Port
Send Delay I/O Percentage (%) are available for all ports.

Figure 2-54 shows a comparison of Zero Buffer Credit Percentage and Port Send Delay I/O
Percentage.

Figure 2-54 Zero Buffer Credit Percentage versus Port Send Delay I/O Percentage

Compression savings %
When using compression on any of the Storwize or SAN Volume Controllers (SVC) devices,
always evaluate whether compression savings are worth the effect of compression on the
system performance. To get a quick view of the compression savings, sort the volumes in
Spectrum Control by compression savings in descending order (1), as shown in Figure 2-55.

Figure 2-55 Volumes sorted by compression savings

44 Managing FlashSystem Performance with IBM Spectrum Control

 For volumes with low compression savings (< 40%), it might be better to turn compression off.
We have seen compressed volumes in the field that were bigger after compression than they
were before.

As described in the IBM Spectrum Control section of IBM Knowledge Center, compression
Savings is calculated by using this formula:
Compression savings % = (1- used space / written space) * 100

2.3.3 Comparing V9000 native tools with IBM Spectrum Control

In IBM Spectrum Control, a comprehensive list of performance metrics is available for
monitoring V9000 performance, as shown in Table 2-2. The V9000 native GUI provides only a
limited set of performance metrics and only on a cumulated/aggregated level per system or
per node.

In Spectrum Control, you can see the performance metrics not only cumulated/aggregated for
the complete storage system or per node, but also for each individual internal resource.

With IBM Spectrum Control, you also can see all the data historically, which is not possible in
the V9000 GUI. The minimum interval in IBM Spectrum Control that can be set to collect
performance data is one minute, whereas the performance chart on the V9000 GUI gets
updated every five seconds. If you need real-time performance data, use the View Real-Time
Performance function, as shown in Figure 2-18 on page 16.

Table 2-2 Performance metrics available in the V9000 GUI

Level Metric Available in
Spectrum Control

System Statistics/Node CPU Utilization: System % X

System Statistics/Node CPU Utilization: Compression % X

System Statistics/Node FC Ports : I/O Rate in ops/s / X

System Statistics/Node FC Ports : Data Rate in MiB/s X

System Statistics/Node iSCSI Ports : I/O Rate in ops/s /

System Statistics/Node iSCSI Ports : Data Rate in MiB/s

System Statistics/Node SAS Ports : I/O Rate in ops/s /

System Statistics/Node SAS Ports : Data Rate in MiB/s

System Statistics/Node ip Replication Ports : I/O Rate in ops/s /

System Statistics/Node ip Replication Ports : Data Rate in MiB/s

System Statistics/Node Volume Read / Write Data Rate in MiB/s X

System Statistics/Node Volume Read / Write I/O Rate in ops/s X

System Statistics/Node Volume Read / Write Latency in ms X

System Statistics/Node MDisk Read / Write I/O Rate in ops/s X

System Statistics/Node MDisk Read / Write Data Rate in MiB/s X

System Statistics/Node MDisk Read / Write Latency in ms X

Chapter 2. Viewing performance information 45

2.4 Monitoring performance for FlashSystem A9000 or A9000R
This section discusses performance monitoring for FlashSystem A9000/A9000R, first using
system native tools and then using Spectrum Control.

2.4.1 Monitoring A9000 or A9000R with HyperScale Manager

FlashSystem A9000 and A9000R GUI uses the IBM Hyper-Scale Manager server. The
management GUI is accessed by directing your web browser at the Hyper-Scale Manager
server (http://<Hyper-ScaleManager server IP address>:8080 or for secure servers
https://<Hyper-ScaleManager server IP address>:8443). A login window, as shown in
Figure 2-56, prompts you to enter valid credentials.

Figure 2-56 IBM Hyper-Scale Manager Login window

Upon successful login, the dashboard is presented. The dashboard lists any Spectrum
Accelerate systems, such as A9000, A9000R, or IBM XIV®, included in the inventory and that
can be accessed through the same credentials. You can select to focus on just one particular
system, as shown in Figure 2-57.

Figure 2-57 Menu to select the system for further actions

46 Managing FlashSystem Performance with IBM Spectrum Control

The selection of a single system changes the dashboard view to display information related to
that particular system only. In the lower right corner, you can see an overview of the system
total IOPS and maximum the latency. This information can be used as a first indication. For
more detailed information, click the Statistics icon to open the Statistics Views menu. The
first four options in that menu pertain to performance statistics:
򐂰 System & Ports Performance Statistics
򐂰 Volume Performance Statistics
򐂰 Host & Port Performance Statistics
򐂰 QoS Performance Statistics

The options are shown in Figure 2-58

Figure 2-58 Different options for performance statistics

If the Volume Performance Statistics is selected, a list of all volumes is created. If Host &
Ports Performance is selected, a list with all hosts is shown. Below the list, there is one chart
for the corresponding selected object and the current IOPS as a metric (see Figure 2-58). For
QoS statistics, historic data is shown.

Based on the selection, all corresponding objects are preselected. For instance, if Volumes
are the monitored object, all volumes belonging to the particular A9000 are selected.

If there is no object selected, no chart is shown. To enable the chart, select at least one object
(Volume, Host, and so on) to get option to open a chart.

Chapter 2. Viewing performance information 47

 Figure 2-59 shows the view with the list of the possible objects that can be monitored and a
single chart with the default settings. You can customize that view. Note that the figure is
specific to Volumes, but the same actions and selections apply to other objects.

You can filter the view to restrict it to a single or small group of volumes. You can enter and
combine different filters.

The columns can be used to have the volumes sorted by size, name, or properties.

The content icon in the top right can be used to customize the columns to be shown.

If the result needs to be saved, the list can be exported into a csv file format.

Selected object

List filter
Selected system

Add or remove the

Selected views chart from window

Figure 2-59 Statistics view for an A9000

The objects that are shown in the chart can now be selected. If there no volume is selected,
no chart is shown. If this is the case, select at least one chart and click the Statistics field at
the bottom of the list, as shown in Figure 2-60.

Figure 2-60 Open a chart for performance statistics

48 Managing FlashSystem Performance with IBM Spectrum Control

The chart with the real-time performance statistics is progressively drawn, and displays the
total values for all selected volumes. This feature allows you to monitor a single volume or a
group of volumes. For example, you can group all objects that belong to an application,
server, or group of servers.

The chart can be further customized by using the pull-down menus in the middle left area of
the window (Figure 2-59 on page 48).

The second menu from the left can be used to select the complete system or only one
interface.

From the third menu, specific metrics can be selected. Because metrics are different at a
System level or host level, the menus are different. The options are as listed here:
򐂰 Qos
– Historic IOPS
– Historic Latency
– Historic Bandwidth
򐂰 Volumes
– Current IOPS
– Historic IOPS
– Current Latency
– Historic Latency
– Current Bandwidth
– Historic Bandwidth
򐂰 Host and Port same as Volumes plus:
– Written by Host
– Written by Host + Forecast
򐂰 System same as Volume plus
– Physical usage
– Physical usage + Forecast
– Written / Allocated
– Written / Projected
– Written / Projected + Forecast
– Projected / Actual

The current metrics are shown for a predefined time frame. For the historic values, there are
many more options to choose from. If one of the historic options is selected, new menus are
visible in this chart. One is the type of I/O, Read, Write, or Read+Write. The second one is
Memory Hit, Memory Miss or Hit+Miss and the third is the size of the I/O or all together.

Any combination of the menu options is possible, like only reads, which have a Memory Miss
and are bigger than 512 KB.

A last menu on the top right of the chart is the Range. The time scale of the chart can be
selected here based on preselected ranges such as last hour or last month, or an individual
time frame can be selected.

Other charts can be added by clicking the Add (+) sign. This action opens a second window
allowing you to compare two different charts. When a chart is no longer needed, it can be
removed with the Close (x) button.

Chapter 2. Viewing performance information 49

 To get more details about the real-time graphs (current) and the values, click in the chart to
open a tooltip window that gives more details about the value at that time. A nice feature for
several monitored volumes (or hosts etc) is the ordering. Depending on the number, the Total
value as well as the max value and min values are shown. Sometimes the average is also
included.

2.4.2 Monitoring A9000 with IBM Spectrum Control

In terms of performance monitoring with IBM Spectrum Control, Flashsystem A9000 and
A9000R are showing basically the same information as for the XIV systems. A complete list of
available performance metrics is available in the IBM Spectrum Control section of IBM
Knowledge Center.

Reporting options are described in 2.5, “IBM Spectrum Control reporting options” on page 55.

Available performance metrics

Performance values are available for the A9000/A9000R grid controllers (interface modules
for XIV), but not for the individual flash modules (MicroLatency Modules) that are listed as
drives or disks in IBM Spectrum Control. No performance metrics are available for the Flash
enclosures that are shown as RAID Arrays in IBM Spectrum Control.

IBM FlashSystem A9000 and IBM FlashSystem A9000R do not track performance statistics
for volumes that were never used. Because there are no performance statistics, performance
metrics are not shown for these volumes and their related components.

Although the IBM Spectrum Control metrics selection offers the SSD Read Cache Hits %
performance metric for the A9000/R as well, they only apply for XIV, which uses solid-state
drives for caching only, as described in the hover help shown in Figure 2-61.

Figure 2-61 A9000 Volume Performance Metric selection

50 Managing FlashSystem Performance with IBM Spectrum Control

2.4.3 Comparing A9000 Hyperscale Manager GUI with IBM Spectrum Control
Table 2-3 lists the performance metrics available in the Hyperscale Manager UI and
IBM Spectrum Control.

Table 2-3 Level of performance metrics

Level Hyperscale Manager UI IBM Spectrum Control

System (per interface/module) X X

Volumes (per interface/module) X X

Domain X

Mapped Host X X

Mirrored Volumes X

Snapshot X

Pool X

Quality of Service X

Port X

With the Hyperscale Manager, the I/O Rate for Memory Hit or Memory Miss can be displayed
as shown on Figure 2-62 as well as the total I/O Rate.

Figure 2-62 Hyperscale Manager: Memory Hit versus Memory Miss I/O Rate

Chapter 2. Viewing performance information 51

 This chart can also be customized for individual blocksizes, or a combination of all blocksizes,
as shown in Figure 2-63. It cannot be combined with other metrics.

Figure 2-63 Hyperscale Manager: I/O Rate for individual Block sizes

In Spectrum Control, the I/O Rate is shown on one y-axis and the Cache Hit Percentage on
the other second y- axis, as illustrated in Figure 2-64.

Figure 2-64 Spectrum Control: I/O Rate versus Cache Hit Percentage

52 Managing FlashSystem Performance with IBM Spectrum Control

 Because a third y-axis cannot be displayed, all three metrics (I/O Rate, Cache Hit
Percentage, Transfer Size) cannot be shown in the same chart (see also Chapter 2.1.2,
“Considerations for Performance Monitoring with IBM Spectrum Control” on page 10).
However, it is possible to show all three metrics at the same time in the table view of the graph
by using the table/chart toggle (1), as shown in Figure 2-65.

Figure 2-65 Spectrum Control table view of performance diagram

As shown in Figure 2-66 and in Figure 2-67 on page 54, two different charts can be displayed
on top of each other with HyperScale Manager. In Spectrum Control, this can be done by
arranging multiple charts accordingly, as described in Figure 2.3.3.

Depending on how many resources are being selected, HyperScale Manager shows
individual performance graphs (Figure 2-66) or accumulated performance graphs
(Figure 2-67 on page 54).

Figure 2-66 Hyper Scale Manager: Individual Performance graphs

Chapter 2. Viewing performance information 53

 Figure 2-67 shows the accumulated performance graphs.

Figure 2-67 Hyper Scale Manager: Accumulated Performance graphs

In Spectrum Control, up to 10 individual graphs can be displayed, as shown in Figure 2-68,

and can be individually toggled on/off as described in 2.1.2, “Considerations for Performance
Monitoring with IBM Spectrum Control” on page 10.

Figure 2-68 Spectrum Control: Up to 10 individual graphs are being displayed

Although IBM Spectrum Control does not show any Quality of Service (QoS) data, volumes
belonging to a QoS could be added to an application as described in IBM Spectrum Family:
IBM Spectrum Control Standard Edition, SG24-8321. Doing so allows performance to be
tracked easily for volumes belonging to a certain QoS.

The minimum interval for collecting performance data with IBM Spectrum Control is
one minute, whereas HyperScale Manager shows real time performance for every
five seconds. Currently, IBM Spectrum Control does not provide the Open Storage System
GUI and View Real Time Performance functionality is not available in IBM Spectrum Control
for A9000/R.

54 Managing FlashSystem Performance with IBM Spectrum Control

2.5 IBM Spectrum Control reporting options
IBM Spectrum Control provides numerous ways of reporting about Flash performance data.
This section provides an overview of the various options and their appropriate usage in
ongoing performance management of IBM FlashSystem storage devices:
򐂰 Web GUI Reports
򐂰 Create Performance Support Package
򐂰 REST API
򐂰 IBM Cognos®
򐂰 Native SQL Reports

2.5.1 Web GUI reports

IBM Spectrum Control Web GUI provides the following export options (see Figure 2-69):
򐂰 To export the data used in the performance chart, click the Export icon located in the
upper left of the chart (1).
򐂰 To export the summary table shown underneath the chart: from the Action pull-down (2),
click More (3) → Export (4) → Select the format (5).
s

3 5
4

Figure 2-69 Performance data export options

2.5.2 Create Performance Support Package

With IBM Spectrum Control v5.2.10, the Create Performance Support Package function
has been improved. The purpose of this function is to provide performance data to the
IBM device support teams. This feature can also be used for other purposes, but it is not
optimized as a generic report generator.

Supported devices are storage systems (both block and file), fabrics, and switches.

Chapter 2. Viewing performance information 55

 Create Performance Support Package using the Web GUI
With IBM Spectrum Control v5.2.10, the Create Performance Support Package function is
implemented in the IBM Spectrum Control GUI. The function can create reports for a specific
time range as a set of compressed CSV files for a single device at a time. Each report is a
single CSV file and covers a single resource type. By default, all resource types supported by
the specified device are included in the zip file.

To create a Performance Support Package for the FlashSystem storage, complete these
steps:
1. Right-click the storage system in the Block Storage Systems window, as shown in
Figure 2-70, and select Create Performance Support Package.

Figure 2-70 Create Performance Support Package in the Web UI

56 Managing FlashSystem Performance with IBM Spectrum Control

2. The Create Performance Support Package window is displayed, as shown in Figure 2-71.
Specify a time range using one of these options:
– One of the predefined ranges (4, 8, or 12 hours) (1)
– Clicking the time range and entering a custom range (2)
For time ranges less than 12 hours, volume data and, if applicable, 1 min interval data are
included in the export files. If you select a time range greater than 12 hours, volume data
and 1 min interval data are included when you select the Advanced Package (3).

3
1
2

Figure 2-71 Specify time-range for the Performance Support Package

3. Click Next (4) and the Spectrum Control server estimates the expected file size, as shown
in Figure 2-72.

Figure 2-72 Expected File size of the Performance Support Package is shown

4. Click Create, as shown in Figure 2-72. Depending on your browser, you might be
prompted to specify where to save the zip file.

Using the Export Performance Package script

There is also a script available to export the performance data. The script, called
exportPerformanceData.bat, is located in the scripts directory of the Spectrum Control
installation directory, as shown in Example 2-6.

Example 2-6 Directory of the exportPerformanceData script

Directory of C:\Program Files\IBM\TPC\scripts

07/21/2017 11:07 PM <DIR> .

07/21/2017 11:07 PM <DIR> ..
07/21/2017 10:45 PM 684 exportPerformanceData.bat
11/09/2016 05:48 PM <DIR> ldap
03/03/2017 05:47 PM <DIR> LTPA
06/02/2017 07:26 PM 4,013 sslCert.bat
07/21/2017 10:57 PM 890 startTPCAlert.bat
07/21/2017 10:55 PM 619 startTPCData.bat
07/21/2017 10:57 PM 850 startTPCDevice.bat
07/21/2017 11:07 PM 817 startTPCWeb.bat
07/21/2017 10:57 PM 1,449 stopTPCAlert.bat

Chapter 2. Viewing performance information 57

 07/21/2017 10:55 PM 616 stopTPCData.bat
07/21/2017 10:57 PM 1,419 stopTPCDevice.bat
07/21/2017 11:07 PM 1,564 stopTPCWeb.bat
04/02/2017 10:26 AM <DIR> TPCNew_Data_201704020126
04/02/2017 10:00 AM 1,289 TPCPerf.bat
11 File(s) 14,210 bytes
5 Dir(s) 341,685,374,976 bytes free

C:\Program Files\IBM\TPC\scripts>

With the script, several parameters can be specified, as shown in Example 2-7.

Example 2-7 exportPerformanceData help

C:\Program Files\IBM\TPC\scripts>exportPerformanceData.bat
Calling Spectrum Control Performance Exporter ...

Usage: exportPerformanceData.bat -user <user> -pwd <password> [options] -resNames <resource_names>

required:
-user The user name for logging into the IBM Spectrum Control server.
-pwd The password for the specified user name.
-resNames One or more storage system, switch, or fabric names as
displayed in the UI, delimited by spaces.
If a name contains spaces, it should be within quotes.
options:
-server Hostname or IP address of the IBM Spectrum Control server.
Default is localhost.
-port Spectrum Control Web server port.
Default is 9569.
-out The name of the exported zip file, or if -zip no is
specified, the directory to store the exported data files.
Default is the current directory.
-resTypes A comma-separated list of resource types for which to export
the performance data.
Valid values are:
storageSystem, nodes, modules, ioGroups, hostConnections
pools, raidarrays, mdisks, disks, volumes, filesystems,
switches, ports, interSwitchConnections
Default is all resource types supported by the specified resource.
-start Start time in the "yyyy-MM-dd HH:mm:ss" format, including quotes.
Overrides -last if both are specified.
-end End time in the "yyyy-MM-dd HH:mm:ss" format, including quotes.
Default is the current time.
-last Length of time range, in hours.
Ignored if -start is also specified. Default is 4 hours, 48 hours, or
720 hours if -summType sample, hourly, or daily is in effect, respectively.
-summType Performance data collection summarization type.
Valid values are: sample, hourly, daily
Default is sample.
-advPkg Include 1 minute samples and volume data if available.
Valid values are: yes, no
Ignored if the length of the specified time range is 12 hours or less.
Default is no unless time range is 12 hours or less.
-zip Compress the CSV files into a zip file.
Valid values are: yes, no
Default is yes.

58 Managing FlashSystem Performance with IBM Spectrum Control

 Example 2-8 shows how to create performance packages including the 1 min interval data
and the volume data for two Flash storage systems by specifying custom start and custom
end dates.

Example 2-8 Create Export Performance Data Package

C:\Program Files\IBM\TPC\scripts>exportPerformanceData.bat -user user -pwd
password -resNames "FlashSystem V9000-9846-V9000_2BB_HS-IBM" "FlashSystem
900-9840-FS900_1-IBM" -start "2016-08-19 08:00:00" -end "2016-08-20 08:00:00"
-advPkg yes

Note: Although multiple devices can be specified in the command, the export performance
package is processed serially for each device.

Export Performance Package

The downloaded Performance Support Package zip file includes one zip file for each device
and each zip file contains csv files for every internal resource type that is supported by the
device, as shown in Figure 2-73.

Figure 2-73 Sample Content of Export Performance Package for a V9000 Storage System

If you open one of the csv files, such as for the resource type volumes, you will find columns
identifying information for each volume, the time stamp, configuration data columns, and
performance data columns, as shown in Figure 2-74.

Figure 2-74 Sample Output of a Volumes.csv File for a V9000 Storage System

Chapter 2. Viewing performance information 59

 The downloaded zip file also contains a log.txt file with many rows per resource type
collected and downloaded. If there were any errors during the data collection or the
download, they will be shown in the log.txt file, as illustrated in Example 2-9.

Example 2-9 log.txt file example

Performance report exported with 'sample' data for block storage system A9000 on
August 18, 2017 10:40:34 AM PDT.
Time range of report: August 18, 2017 6:40:24 AM PDT with a duration of 4 hours 0
minutes.

Storage System data: 240 rows were processed.

Port data: 1,440 rows were processed.
Host Connection data: 240 rows were processed.
Pool data: 960 rows were processed.
Volume data: 61,696 rows were processed.
Module data: 723 rows were processed.

Note: If you create performance packages for multiple devices, you might run into NTFS
restrictions while unzipping the files and receive a “path too long” error. In this situation,
either move the files to a directory with a shorter path or rename the files to reduce the
length of the path name.

2.5.3 REST API

Although the Representational State Transfer (REST) API implementation for Spectrum
Control is not a reporting tool, it can be used to access information about resources in json
format.

Using the browser:

The IBM Spectrum Control REST API is hosted here:
https://<hostname>:9569/srm/REST/api/v1/

To get a list of StorageSystems with their configuration data, specify the following URL:
https://<hostname>:9569/srm/REST/api/v1/StorageSystems

A sample output result is shown in Example 2-10.

Example 2-10 Example output of a REST request for Storage Systems

[
{
"Acknowledged": "No",
"Allocated Space": "8,830.84",
"Assigned Volume Space": "100",
"Available Pool Space": "685,792.00",
....
"Firmware": "8.1.0.0 (build 137.3.1708160754000)",
"Last Successful Monitor": "Aug 17, 2017, 20:40:00",
"Last Successful Probe": "Aug 16, 2017, 21:21:03",
....
"Model": "DH8",
"Name": "ssd-svc-dh8-01",
....

60 Managing FlashSystem Performance with IBM Spectrum Control

 "Type": "SAN Volume Controller - 2145",
"id": "7494"
},
{
"Acknowledged": "No",
"Allocated Space": "22,994.68",
....
"FlashCopy": "30",
"IP Address": "9.155.112.248",
....
"Raw Disk Capacity": "64,820.05",
"Read Cache": "452.56",
"Remote Relationships": "331",
"Serial Number": "75FAW81",
"Shortfall": "27",
"Status": "Normal",
....
"Type": "DS8000",
"Unassigned Volume Space": "5,555.81",
"Unprotected Volumes": "1567",
"Unreserved Pool Space": "24,245.00",
"Used Pool Space": "24,003.00",
"Used Space": "22,994.68",
"Vendor": "IBM",
"Virtual Allocation": "49.31",
"Volumes": "1,861",
"Write Cache": "32.00",
"id": "7540"
},
....
}
]

Adding the StorageSystem id and a resource type, for example Volumes, to the URL will
display a list of volumes for that storage system:
https://<hostname>:9569/srm/REST/api/v1/StorageSystems/<StorageSystem id>/Volumes

To get a list of available performance metrics for a StorageSystem, add Volume id

/Performance in the URL:
https://<hostname>:9569/srm/REST/api/v1/StorageSystems/<StorageSystem
id>/Volumes/<Volume id>/Performance

You can also specify a specific metric id:

https://<hostname>:9569/srm/REST/api/v1/StorageSystems/<StorageSystem
id>/Volumes/<Volume id>/Performance/<metric id>

Example 2-11 shows the response times of a A9000 Volume in json format.

Example 2-11 Using RESTAPI to get performance data

https://9.155.122.36:9569/srm/REST/api/v1/StorageSystems/65100/Volumes/2160532/Per
formance/823

[
{

Chapter 2. Viewing performance information 61

 "metricDetails": {
"823": {
"description": "Average number of milliseconds that it took to service
each write operation, for a particular component over a particular time
interval.",
"name": "Write Response Time",
"units": "ms\/op"
}
}
},
{
"componentId": 2160532,
"current": [
{
{
"x": 1503016447000,
"y": "0.00"
},
},
{
"x": 1503016747000,
"y": "16.07"
},
{
"x": 1503016807000,
"y": "16.07"
},
{
"x": 1503016867000,
"y": "16.73"
},
{
"x": 1503016928000,
"y": "16.48"
},
{
"x": 1503016987000,
"y": "16.69"
},
{
"x": 1503017047000,
"y": "16.01"
},
{
"x": 1503017107000,
"y": "16.25"
},

Multiple resources and multiple metrics can be queried at the same time by using the
following syntax:
https://<hostname>:9569/srm/REST/api/v1/StorageSystems/<StorageSystem
id>/Volumes/Performance?<metric id1>,<metric id2>,<metric id3>&ids=<Volume
id1>,Volume id3>,Volume id3>

62 Managing FlashSystem Performance with IBM Spectrum Control

 A sample output is shown in Example 2-12.

Example 2-12 Querying multiple metrics and multiple resources

https://9.155.122.36:9569/srm/REST/api/v1/StorageSystems/65100/Volumes/Performance?metrics=823,8
22,821&ids=2160532,2160549,2160521

[
{
"metricDetails": {
"821": {
"description": "Average number of mebibytes (2^20 bytes) per second transferred for
read and write operations.",
"name": "Total Data Rate",
"units": "MiB\/s"
},
"822": {
"description": "Average number of milliseconds that it took to service each read
operation, for a particular component over a particular time interval.",
"name": "Read Response Time",
"units": "ms\/op"
},
"823": {
"description": "Average number of milliseconds that it took to service each write
operation, for a particular component over a particular time interval.",
"name": "Write Response Time",
"units": "ms\/op"
}
}
},
{
"componentId": 2160521,
"current": [
{
"x": 1502985600000,
"y": null
},
{
"x": 1502987827000,
"y": "0.32"

....
{
"x": 1503075600000,
"y": null
}
],
"deviceId": 65100,
"deviceName": "AIX_BIG_001<br \/> A9000",
"endTime": 1503075600000,
"label": "Write Response Time",
"maxValue": 40.56,
"metricId": 823,
"minValue": 0.0,
"precision": 2,

Chapter 2. Viewing performance information 63

 "resourceID": 2160521,
"startTime": 1502989200000,
"units": "ms\/op"
},
{
"componentId": 2160521,
"current": [
{
"x": 1502989200000,
"y": null
},
{
"x": 1502989327000,
"y": "0.20"

....
],
"deviceId": 65100,
"deviceName": "AIX_BIG_001<br \/> A9000",
"endTime": 1503075600000,
"label": "Read Response Time",
"maxValue": 101.56,
"metricId": 822,
"minValue": 0.0,
"precision": 2,
"resourceID": 2160521,
"startTime": 1502989200000,
"units": "ms\/op"
}, },
{
"componentId": 2160532,
"current": [
{
"x": 1502989200000,
"y": null
},
{
"x": 1502989327000,
"y": "0.00"

Tip: The timestamps are shown in epoch time. To convert the timestamps, use the
EpochConverter.

The granularity as well as start and end time for the performance data can be specified as
well:
https://<server>:9569/srm/REST/api/v1/StorageSystems/<id>/Volumes/<vol
id>/Performance/824?granularity=sample&startTime=1471935241000&endTime=14720215680
00

Note: The REST API syntax is case sensitive.

Reference: For more information regarding the implementation of the RESTful API in IBM
Spectrum Control, see IBM Knowledge Center.

64 Managing FlashSystem Performance with IBM Spectrum Control

 Using Wget
GNU Wget is a free software package that retrieves content from web servers using HTTP,
HTTPS, and FTP and is part of the GNU Project.

Reference: wget can be downloaded from the GNU website.

For authentication, a security token can be stored in a file, such as cookies.txt for later
queries, as shown in Example 2-13.

Example 2-13 Authentication token

wget --post-data "j_username=<username>&j_password=<password>" --no-check-certificate
--keep-session-cookies --save-cookies cookies.txt
https://<servername>:9569/srm/j_security_check

Now this security token can be used for queries against the IBM Spectrum Control’s REST
API, as illustrated in Example 2-14.

Example 2-14 Retrieving the authentication token

wget --no-check-certificate --load-cookies cookies.txt -O Output.html
https://<servername>:9569/srm/REST/api/v1/StorageSystems/<id>/Volumes

In Example 2-15, the previously saved security token in cookies.txt is being used for the
authentication. The output of this request is then saved in the file Output.html, but still in the
JSON format.

To parse the JSON output, a tool like JSON query (jq) can be used.

Reference: JSON Query can be obtained from the GitHub website.

Example 2-15 Parsing the json output file

C:\Users\IBM_ADMIN>type C:\SpectrumControl\wget\Output.html |jq -r
".[]|[.id,.Name,.Capacity,.\"Storage System\"]|@csv"
"2160667","AIX_HS_027","45.63","A9000"
"2160664","AIX_HS_024","45.63","A9000"
"2160665","AIX_HS_025","45.63","A9000"
"2160666","AIX_HS_026","45.63","A9000"
"2160668","AIX_HS_028","45.63","A9000"
"2160669","AIX_HS_029","45.63","A9000"
"2160670","AIX_HS_030","45.63","A9000"
"2160671","AIX_HS_031","45.63","A9000"
"2160672","AIX_HS_032","45.63","A9000"
"2160673","AIX_HS_033","45.63","A9000"
"2160675","AIX_HS_035","45.63","A9000"

To get rid of the quotation marks and backslashes, the jq command can be combined with the
sed utility, as illustrated in Example 2-16.

Example 2-16 The sed utility

C:\Users\IBM_ADMIN>type C:\SpectrumControl\wget\Output.html |jq -r
".[]|[.id,.Name,.Capacity,.\"Storage System\"]|@csv" |sed -e "s/\"*//g"
2160667,AIX_HS_027,45.63,A9000

Chapter 2. Viewing performance information 65

 2160664,AIX_HS_024,45.63,A9000
2160665,AIX_HS_025,45.63,A9000
2160666,AIX_HS_026,45.63,A9000
2160668,AIX_HS_028,45.63,A9000
2160669,AIX_HS_029,45.63,A9000
2160670,AIX_HS_030,45.63,A9000
2160671,AIX_HS_031,45.63,A9000
2160672,AIX_HS_032,45.63,A9000
2160673,AIX_HS_033,45.63,A9000
2160674,AIX_HS_034,45.63,A9000

2.5.4 IBM Cognos

In IBM Spectrum Control, it is possible to run and view predefined Cognos reports and create
custom reports. The Cognos reporting engine is accessible using the Reports action in the
top navigation bar.

IBM Spectrum Control provides over 70 predefined reports that display capacity and
performance information collected by IBM Spectrum Control. Charts are automatically
generated for most of the predefined reports. Depending on the type of resource, the charts
show statistics for space usage, workload activity, bandwidth percentage, and other statistics.
You can schedule reports and specify to create the report output in HTML, PDF, and other
formats. You can also configure reports to save the report output to your local file system or to
send reports as mail attachments.

FlashSystem 840/FlashSystem 900 Performance data are not included in the performance
package. Therefore, it is not possible to get any performance data for the FlashSystem 840/
FlashSystem 900 using Predefined reports or using the Performance package for custom
reports.

However, you can use native SQL for creating queries against the FlashSystem 840/
FlashSystem 900 performance tables. Native SQL Queries can be rendered with IBM Cognos
Report Studio.

Reference: For more information regarding the usage of Cognos and its functions, see
IBM Tivoli Storage Productivity Center V5.1 Technical Guide, SG24-8053, and IBM
Spectrum Family - Spectrum Control Standard, SG24-8321.

2.5.5 Native SQL reports

IBM Spectrum Control stores all data in a DB2 database called TPCDB that you can query
easily by using native SQL.

Connections can also be done using the ODBC interface, for example with Microsoft Excel, or
you can render your native SQL Query in IBM Cognos by using IBM Cognos Report Studio.

Tip: Always specify with ur for read only in your SQL Queries. Otherwise your tables
might get locked during the read operation and slow down the TPCDB performance.

For more information see: Locks and concurrency control and read-only-clause at IBM
Knowledge Center.

66 Managing FlashSystem Performance with IBM Spectrum Control

 Performance data for the FlashSystem 840 / FlashSystem 900 are stored in the following DB2
tables:
򐂰 Sample performance data for flash modules, nodes, ports, and the entire subsystem:
– T_PRF_FLSH_MODULE
– T_PRF_FLSH_NODE
– T_PRF_FLSH_PORT
– T_PRF_FLSH_SYSTEM
򐂰 Hourly and daily performance data for flash modules, nodes, ports, and the entire
subsystem:
– PRF_T_FLSH_MODULE
– T_PRF_T_FLSH_NODE
– T_PRF_T_FLSH_PORT
– T_PRF_T_FLSH_SYSTEM

An example of querying the Spectrum Control database using native SQL with Microsoft
Excel, and an example of how to render a native SQL Query in IBM Cognos is provided in
Chapter 5 of IBM Spectrum Family - Spectrum Control Standard, SG24-8321.

2.5.6 TPCTOOL
You can use the TPCTOOL command-line interface (CLI) to extract data from the IBM
Spectrum Control database. It requires no knowledge of the IBM Spectrum Control schema
or SQL query skills, but you need to understand how to use the tool. For more information
regarding the TPCTOOL, see these publications:
򐂰 IBM Tivoli Storage Productivity Center V5.2 Release Guide, SG24-8204
򐂰 Reporting with TPCTOOL, REDP-4230

To get performance data for FlashSystem 900,you can use TPCTOOL as well. Proceed as
follows:
1. Start a command window (CMD).
2. From the <install>/cli directory enter tpctool -user db2admin -pwd password to get
the tpctool prompt.
3. Enter lsdev -perf -l to get the GUID of devices for which performance data is collected
as shown in Example 2-17.

Example 2-17 Using TPCTOOL

C:\Program Files\IBM\TPC\cli>tpctool -user db2admin -pwd password
tpctool> lsdev -perf -l
GUID Name Label Type Status Timestamp
==========================================================================================================================================
=======
100000051E0B2F42 100000051E0B2F42 Blueswitch-1 UNREACHABLE
2017.07.06:13:26:22
0000020060C14298+0 SVC-2145-svc_72_73-IBM SVC-2145-svc_72_73-IBM IBM SAN Volume Controller NORMAL
2017.08.18:19:48:29
00000200A1200082+0 Storwize V7000-2076-v7000_ctr_5-IBM v7000_ctr_5 IBM Storwize V7000 NORMAL
2017.08.17:21:38:16
2107.75ACA91+0 DS8000-2107-75ACA91-IBM 2107.75ACA91 IBM System Storage DS8000 NORMAL
2017.08.18:04:45:24
2810.1340008+0 XIV-2810-XIV_04_1340008-IBM XIV_04_1340008 IBM System Storage XIV NORMAL
2017.08.18:06:24:45
00000200A1C004EA+0 Storwize V7000-2076-v7000_ctr_04-IBM v7000_ctr_04 IBM Storwize V7000 NORMAL
2017.08.18:05:47:07
0000020063A20412+0 FlashSystem 840-9840-FlashSystem-840-03-IBM FlashSystem-840-03 IBM FlashSystem 840 NORMAL
2017.08.17:21:15:41
2810.1340010+0 XIV-2810-XIV_PFE02_1340010-IBM XIV_PFE02_1340010 IBM System Storage XIV NORMAL
2017.08.18:06:31:04

Chapter 2. Viewing performance information 67

9835.1320902+0 FlashSystem A9000-9835-A9000R-IBM A9000R FlashSystem A9000R NORMAL
2017.08.18:12:59:26
9836.1322131+0 FlashSystem A9000-9836-A9000-IBM A9000 FlashSystem A9000 NORMAL
2017.08.17:21:44:36
00000203224045BC+0 FlashSystem V9000-9846-Cluster_Hotel-IBM PFE_V9000(Cluster_Hotel) IBM FlashSystem V9000 NORMAL
2017.08.17:22:03:21
tpctool>

4. With lstype, you get a list of available components, as shown in Example 2-18.

Example 2-18 Output of lstype command

tpctool> lstype
Name Type Description
====================================
subsystem 1 Subsystem
subsys_port 2 "HBA port"
controller 3 Controller
host_conn 4 "Host Connection"
svc_iogrp 5 "SVC I/O Group"
ds_rio 6 "RIO Loop"
stor_pool 7 "Storage Pool"
da 8 "Device Adapter"
ds_rank 9 Rank
array 10 Array
svc_mdisk 11 "SVC Managed Disk"
vol 12 Volume
switch 13 Switch
switch_port 14 "Switch Port"
svc_node 15 "SVC Node"
svc_disk 20 Disk
switch_trunk 27 "Switch Trunk"

5. Use lsmetrics shown in Example 2-19 to get the available performance metrics for the
storage subsystem ports of FlashSystem 900.

Example 2-19 Output of lsmetrics command

tpctool> lsmetrics -subsys 0000020063A20412+0 -ctype subsys_port
Metric Value
==============================================
"Port Send Data Rate" 858
"Port Receive Data Rate" 859
"Port Send I/O Rate" 852
"Port Receive I/O Rate" 853
"Total Port I/O Rate" 854
"Total Port Data Rate" 860
"Port to Host Send I/O Rate" 901
"Port to Host Receive I/O Rate" 902
"Total Port to Host I/O Rate" 903
"Port to Disk Send I/O Rate" 904
"Port to Disk Receive I/O Rate" 905
"Total Port to Disk I/O Rate" 906
"Port to Local Node Send I/O Rate" 907
"Port to Local Node Receive I/O Rate" 908
"Total Port to Local Node I/O Rate" 909
"Port to Remote Node Send I/O Rate" 910
"Port to Remote Node Receive I/O Rate" 911
"Total Port to Remote Node I/O Rate" 912

68 Managing FlashSystem Performance with IBM Spectrum Control

 "Port to Host Send Data Rate" 913
"Port to Host Receive Data Rate" 914
"Total Port to Host Data Rate" 915
"Port to Disk Send Data Rate" 916
Press Enter To Continue...

6. The getrpt command as shown in Example 2-20 lists the port sample performance data.

Example 2-20 Output of getrpt command

tpctool> getrpt -subsys 0000020063A20412+0 -ctype subsys_port -columns 852,853,854 -level sample
-duration 3600 -start 2017.08.17:08:00:00 -fs ;
Timestamp;Interval;Device;Component;852;853;854
===============================================
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 1, Adapter 1, Port 1";1264;1262;2526
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 1, Adapter 1, Port 2";1264;1262;2526
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 1, Adapter 1, Port 3";0;0;0
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 1, Adapter 1, Port 4";0;0;0
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 1, Adapter 2, Port 1";1262;1264;2526
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 1, Adapter 2, Port 2";1264;1262;2526
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 1, Adapter 2, Port 3";0;0;0
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 1, Adapter 2, Port 4";0;0;0
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 2, Adapter 1, Port 3";0;0;0
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 2, Adapter 1, Port 4";0;0;0
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 2, Adapter 2, Port 3";0;0;0
2017.08.17:08:00:17;302;FlashSystem-840-03;"Canister 2, Adapter 2, Port 4";0;0;0
2017.08.17:08:05:19;301;FlashSystem-840-03;"Canister 1, Adapter 1, Port 1";1242;1250;2492
2017.08.17:08:05:19;301;FlashSystem-840-03;"Canister 1, Adapter 1, Port 2";1245;1246;2492
2017.08.17:08:05:19;301;FlashSystem-840-03;"Canister 1, Adapter 1, Port 3";0;0;0
2017.08.17:08:05:19;301;FlashSystem-840-03;"Canister 1, Adapter 1, Port 4";0;0;0
2017.08.17:08:05:19;301;FlashSystem-840-03;"Canister 1, Adapter 2, Port 1";1244;1247;2492
2017.08.17:08:05:19;301;FlashSystem-840-03;"Canister 1, Adapter 2, Port 2";1246;1245;2492
2017.08.17:08:05:19;301;FlashSystem-840-03;"Canister 1, Adapter 2, Port 3";0;0;0
2017.08.17:08:05:19;301;FlashSystem-840-03;"Canister 1, Adapter 2, Port 4";0;0;0
2017.08.17:08:05:19;301;FlashSystem-840-03;"Canister 2, Adapter 1, Port 3";0;0;0
2017.08.17:08:05:19;301;FlashSystem-840-03;"Canister 2, Adapter 1, Port 4";0;0;0
Press Enter To Continue...

Chapter 2. Viewing performance information 69

70 Managing FlashSystem Performance with IBM Spectrum Control
 3

Chapter 3. Setting thresholds and alerts

This chapter describes the following concepts:
򐂰 Defining alerts and thresholds in IBM Spectrum Control
򐂰 Alert and threshold notifications
򐂰 Guidelines for thresholds

© Copyright IBM Corp. 2018. All rights reserved. 71

3.1 Defining alerts and thresholds in IBM Spectrum Control
IBM Spectrum Control provides multiple possibilities to set up alerts and thresholds to help
detect errors at an early stage and alert those responsible. IBM Spectrum Control also
provides various options for being notified when a specified threshold was exceeded (see 3.2,
“Alert and threshold notifications” on page 82) and how to suppress alerts so that you are not
alerted more often than necessary. (see 3.2.6, “Alert suppressions” on page 92).

Alerts/thresholds can be defined at different levels:

򐂰 Device level
򐂰 Application level
򐂰 General group level

3.1.1 Alerts and thresholds at device level

Refer to Figure 3-1. Alerts and thresholds can be set up for each device, individually on the
device itself, or on its internal resources (1). Alerts can be set on general conditions, and
thresholds can be defined for capacity and performance metrics (2). Alerts can be enabled
and disabled by using the switch icon (3). The severity of alerts can be adjusted (4).

4
3

Figure 3-1 IBM Spectrum Control Alert Definition window

Performance thresholds can be defined for every performance metric that can be selected in
the performance chart for the selected resource.

72 Managing FlashSystem Performance with IBM Spectrum Control

 In addition to maximum or minimum threshold values (1), threshold ranges with different
severities can be defined (2), as shown in Figure 3-2.

Figure 3-2 Performance Thresholds

Starting with IBM Spectrum Control version 5.2.11, multi-metric, multi-condition, and
multi-component thresholds can be defined by using the Custom tab.

With this function, you can define up to five attributes that all need to be true to trigger a
threshold violation alert, as shown in Figure 3-3. In this example, an alert is only triggered if
the read response time is higher than 22 ms and the Read I/O Rate is higher than 100 ms/op
for volumes that are thin provisioned.

&
&

&
&

Figure 3-3 Multimetric Alert Definition in Spectrum Control

Chapter 3. Setting thresholds and alerts 73

 As shown in Figure 3-4, the combination of multi-components with multi-metrics is also
possible using custom alert definitions.

In this example, the alert is only triggered if the read I/O rate is in the range 1,000 - 5,000
ops/s and the read response time is higher than 20 ms/op and the Pool Activity Score (which
is a synonym to Access Density) is higher than 0.7 IOPS/GiB.

&
&
&

Figure 3-4 Multicomponent Alert Definition in Spectrum Control

Note: Detailed instructions on how to set up custom alerts can be found in the IBM
Spectrum Control section of IBM Knowledge Center.

3.1.2 Alerts and thresholds for applications

Volumes in the same storage system might not all have the same performance requirements
or performance characteristics, especially when using SAS enclosures in a V9000. Therefore,
it might be necessary to set up different thresholds for different volumes or group of volumes.

Similar grouping can be done for other resources, as listed here:

򐂰 Applications
򐂰 Data stores
򐂰 Hypervisors
򐂰 File Systems (Only file systems that are monitored through a Storage Resource Agent.)
򐂰 File sets
򐂰 Servers
򐂰 Shares
򐂰 Vaults (IBM Cloud Object Storage)
򐂰 Virtual Machines
򐂰 Volumes
򐂰 Volume Groups

There are several options on how you can add resources, such as volumes, to an application:
򐂰 Using filters to assign resources to applications
For more information, see the Spectrum Control section of IBM Knowledge Center.
򐂰 Adding resources manually to applications, which is possible using these methods:
– Adding resources with the command-line interface
– Adding resources individually with the command-line interface
– Adding resources using bulk assignment
For more information, see the IBM Spectrum Control section of IBM Knowledge Center.

74 Managing FlashSystem Performance with IBM Spectrum Control

 The various options for creating applications are also described in section 5.2 of IBM
Spectrum Family: IBM Spectrum Control Standard Edition, SG24-8321.

After creating applications, alert definitions can be set for the members of an application as
described in 3.1.1, “Alerts and thresholds at device level” on page 72 and shown in
Figure 3-5.

Figure 3-5 Alert definition at application level

Custom Alerts can also be created at an application level.

Chapter 3. Setting thresholds and alerts 75

 A hierarchy of applications can be built so that one or more subcomponents can be added to
an application, as shown in Figure 3-6.

Figure 3-6 Application with Subcomponents

Alerts can be defined on the main applications as well and are valid for all subcomponents
and their components, as shown in Figure 3-7.

Figure 3-7 Custom alert definition for Application with Subcomponents

76 Managing FlashSystem Performance with IBM Spectrum Control

 Considerations for creating applications
Some restrictions apply on how you can add certain resources:
򐂰 VMs cannot be added directly to an application using the GUI. Instead, you must add the
VM’s agentless servers1 to applications or use the CLI.
򐂰 Vaults can only be added to applications by using the GUI.

The option of using filters to assign resources to applications provides the advantage that the
created application becomes self-maintaining. Whenever a volume/resource is being created
that matches the defined filter criteria, the volume/resource is automatically added to the
application. If filters are not used, the newly created volumes/resources need to be added
manually to the application.

If multiple filters are being used to assign resources to an application, then resources that
comply to at least one of the filters are added to the application. Multiple filters are combined
with the OR conjunction, as shown in Figure 3-8.

Figure 3-8 Using multiple filters to assign resources to an application

Currently, backend volumes that are virtualized by an SVC or volumes used for the IFS file
system within a V7000 Unified are intentionally excluded when filters are used.

1 Agentless servers are automatically created starting with IBM Spectrum Control v5.2.15.

Chapter 3. Setting thresholds and alerts 77

3.1.3 Alerts and thresholds for general groups
Creating groups with resources others than listed for applications can be done by adding
them to a general group. All kinds of resources can be added to a general group as described
in the General Groups window in IBM Spectrum Control:
1. Add a resource to a general group from the page where that resource is listed. For
example, to add a block storage system, go to the Block Storage Systems window,
right-click a storage system, and select Add to General Group, as shown in Figure 3-9.

Figure 3-9 Adding Ports to General Group

78 Managing FlashSystem Performance with IBM Spectrum Control

 2. Alerts/Thresholds can then be defined as for applications for the members of the general
group, as shown in Figure 3-10.

Figure 3-10 General Group Alert definition for A9000 ports

3. To avoid having to specify alerts/thresholds multiple times for the same kind of storage
systems, a general group can be created. See Figure 3-11 for an example for
FlashSystem A9000.

Figure 3-11 Adding multiple Storage systems to a general group

Chapter 3. Setting thresholds and alerts 79

 4. To be able to set up alerts/thresholds on the internal resources of the storage systems, the
internal resources need to be added separately to the same general group, as shown in
Figure 3-12.

Figure 3-12 Adding internal resources to an existing general group

80 Managing FlashSystem Performance with IBM Spectrum Control

 5. Now all thresholds can be specified on the various members of the general group
including custom alerts for multiple components for all resources of the same kind.
In the members section of Figure 3-13 (1) you can see that two A9000 systems, 404
volumes, and 10 pools were added as members to the A9000 GG General Group.
Alert definitions can be specified for all kinds of members (2) and you can see how many
alert definitions have already been configured and enabled. For instance, in Figure 3-13,
one alert has been configured at the storage system level, one at port level, and one
custom alert has been set up.

Figure 3-13 Alert definitions for multiple A9000 Storage Systems

General groups can be nested like applications.

Chapter 3. Setting thresholds and alerts 81

3.1.4 Considerations about using applications and general groups
The significant advantage of using applications is that they can be created by using the filter
functionality and are therefore self-maintaining.

Applications also have nice overview windows where capacity data and performance data are
displayed, as shown in Figure 3-14.

Figure 3-14 Application Overview window

Another advantage of using applications is that they also can be used for chargeback and
storage consumer reports as described in the IBM Spectrum Control section of
IBM Knowledge Center.

Although you only can add a few resources to applications (see Appendix 3.1.2, “Alerts and
thresholds for applications” on page 74), general groups have the advantage that all kinds of
resources can be added. This capability includes complete storage systems with all their
internal resources. Therefore, alerts need to be defined only once at a general group level
rather than configuring alerts for each storage system individually.

Because IBM Spectrum Control already provides default alerts at a device level, it might be
necessary to disable them if you create similar alerts at an application or general group level

3.2 Alert and threshold notifications

Alert violations can be seen on various windows in IBM Spectrum Control, including these
locations:
򐂰 General Alert tab where all alerts can be seen
򐂰 Alert tab of each device
򐂰 Alert tab of each application
򐂰 Alert tab of each of the general groups

82 Managing FlashSystem Performance with IBM Spectrum Control

 An individual alert is shown in Figure 3-15. It shows the actual value (1), the alert condition
(2), the (internal) resource that triggered the alert (3), the storage system to which the internal
resource belongs (4) and, during a performance threshold violation, the performance chart
with data before and after the threshold violation (5) including the defined threshold (6). The
threshold violation can be easily detected (7). The chart can be opened in a new window by
clicking the Open New Window icon (8).

2 1 3

5 7 6 8

Figure 3-15 Threshold violation window

The affected servers can also be easily detected, as shown in Figure 3-16.

Figure 3-16 Affected Servers

Chapter 3. Setting thresholds and alerts 83

 Alerts can be acknowledged, exported, and removed, as shown in Figure 3-17.

Figure 3-17 Alert actions

Notifications can be configured at these different levels:

򐂰 General setting (for email only)
򐂰 Per device
򐂰 Per group
– Per application
– Per subcomponent
– Per general group
– Per subgroup
򐂰 Per individual alert

84 Managing FlashSystem Performance with IBM Spectrum Control

 For each of the above specified levels, it is possible to define the Notification action (1),
which can be any of the following actions, as shown in Figure 3-18 (2):
򐂰 Create an entry in the Windows log
򐂰 Send an email
򐂰 Send an SNMP trap
򐂰 Send a notification to Netcool/OMNIbus
򐂰 Trigger a script based on an alert

Figure 3-18 Alert configuration

3.2.1 Event notification in Windows log

If configured as shown in Figure 3-18, alert notifications are written to the Windows event log,
as shown in Figure 3-19.

Figure 3-19 Alert notification in Windows log

Chapter 3. Setting thresholds and alerts 85

 Detailed information is available for each event logged.

3.2.2 Event notification to be sent by email

A prerequisite for having event notifications being sent by email is the configuration of the
email server. To configure the email server, click Settings → Alert Notifications → Email, as
shown in Figure 3-20.

2 : Configure global email notification recipient

Figure 3-20 Alert Notifications using email

Here the email server can be configured, including authentication credentials (optional) (1). A
Global email notification recipient can be set to receive all emails (2).

86 Managing FlashSystem Performance with IBM Spectrum Control

 The recipient for alerts can be overridden for each individual alert definition using the email
notification icon (1), selecting Override notification settings (2), and specifying the recipient
for this specific alert definition (3), as shown in Figure 3-21.

1 2
3

Figure 3-21 Override notification settings

Notification settings can also be overridden at a device or group level using the Notification
Settings tab, as shown in Figure 3-22.

Figure 3-22 Notification Settings

Note: The event notification set at a more detailed level overrides the event notification set
at a higher level within the same type. For example, the alert notification definition for an
individual alert on an internal resource of a storage system overrides the alert notification
definition at the storage system level. Likewise, the alert notification definition of a
subcomponent overrides the alert notification definition of an application.

Chapter 3. Setting thresholds and alerts 87

 Security standards of the email server are automatically detected. The email server is
automatically trusted, so no additional tasks are required.

When an email is sent, IBM Spectrum Control attempts to send it using various connection
and encryption mechanisms, from most to least restrictive:
򐂰 Attempt Secure Sockets Layer connection from the outset with data always encrypted
(usually using port 465)
򐂰 Attempt TLS encryption that can be started by STARTTLS command at SMTP level if the
server supports it (usually using port 567)
򐂰 If both of the above techniques fail, attempt unsecured connection (usually using port 25)

IBM Spectrum Control now accepts all server certificates. This feature allows the
administrator to connect to a restricted SMTP server without downloading and importing
server certificates into the certificate store for each IBM Spectrum Control server.

The email alert notification shows the condition that was violated (1), the actual value (2), the
internal resource (3), and the device to which the internal resource belongs (4). In addition,
hyperlinks to the internal resource’s properties book (5) and to the alert (6) are included, as
shown in Example 3-1.

Example 3-1 Email alert notification sample

Alert Read Response Time >= 3 ms/op & < 5 ms/op has been triggered. 1

Violation: 3.22 ms/op 2

Internal Resource: AIX_HS_090, 3
View Resource =
https://vm12236.mainz.de.ibm.com:9569/srm/gui#alertResourceRedirector?parent.type=
storageSystem&parent.id=65100&type=volume&id=2160730 5
Resource: A9000 4

View Alert = https://vm12236.mainz.de.ibm.com:9569/srm/gui#alerts?id=161367 6

88 Managing FlashSystem Performance with IBM Spectrum Control

3.2.3 SNMP
Alert notifications can also be sent by SNMP trap to one or two SNMP destinations, which can
be configured by selecting Settings → Alert Notifications → SNMP.

A second SNMP trap receiver can be configured by using the Add SNMP Destination 2
option, as shown in Figure 3-23.

Figure 3-23 Configure SNMP trap receiver

To receive SNMP traps, the SNMP trap receiver must be configured with the appropriate
Management Information Base (MIB) file.

The MIB file can be found as specified in Table 3-1.

Table 3-1 Locating the MIB file

Location of MIB file Directory

Installation media unzip_dir\data\snmp\tivoliSRM.mib

After IBM Spectrum Control is install_dir\data\snmp\tivoliSRM.mib

installed

Chapter 3. Setting thresholds and alerts 89

3.2.4 Netcool/OMNIbus
Alert notifications can also be sent to a Netcool/OMNIbus server, which can be specified in
Settings → Alert Notifications → Netcool/OMNIbus, as shown in Figure 3-24.

Figure 3-24 Netcool/OMNIbus configuration in IBM Spectrum Control

The Netcool/OMNIbus server must be configured to handle alert notifications sent by IBM
Spectrum Control. It requires the Event Integration Facility (EIF) rule files that can be
obtained from the IBM Spectrum Control server, as listed in Table 3-2.

Table 3-2 Event Integration Facility (EIF)

Rules file names Location of rules files in IBM Spectrum Control Copy rules files to this
location in IBM Tivoli®
Netcool/OMNIbus

tivoli_eif_tpc.rules extracted_installation_image/data/omnibus $OMNIHOME/probes/arch/

tivoli_eif_tpc_tbsm.rules

Reference: For more information, see IBM Knowledge Center, at Configuring Tivoli
Netcool/OMNIbus alert notifications.

90 Managing FlashSystem Performance with IBM Spectrum Control

3.2.5 Trigger a script based on an alert
After an alert was triggered, scripts can be run from any server where a storage resource
agent (SRA) is deployed (including the IBM Spectrum Control server). After clicking the Alert
notification icon (1), select Run script, and then Upload script (3). The available
parameters can be passed on to a script are listed (4), as shown in Figure 3-25.

1 3

Figure 3-25 Triggered Run script functionality

The script file MUST be in either of these subdirectories:

򐂰 <install_dir>\IBM\TPC\data\scripts
򐂰 <install_dir>\IBM\TPC\agent\scripts

Chapter 3. Setting thresholds and alerts 91

 It might be necessary to Enable running scripts on agent on the server’s SRA, as shown in
Figure 3-26.

Figure 3-26 Enable running scripts on agent

3.2.6 Alert suppressions

To avoid getting unnecessary alerts, you can proactively suppress alerts that you do not
need.

Four different settings can be specified after selecting the Alert Removal icon, as shown in
Figure 3-27.

Figure 3-27 Suppression Settings in IBM Spectrum Control

򐂰 Option 1: Do not suppress alerts:

All alerts / threshold violations will generate the specified notification. This setting means
that if the performance data is collected every minute, the notifications are sent out every
minute while the threshold violation persists.

92 Managing FlashSystem Performance with IBM Spectrum Control

򐂰 Option 2: Only alert once until problem clears (default setting)
The notification is triggered at the beginning of the threshold violation (1), and no
notifications are sent while the threshold violation persists (2). If the threshold violation
recurs (3) after it had been cleared (4), the notification will be sent out again, as shown in
Figure 3-28.

1 2 3

Threshold

2 4 2

Figure 3-28 Only alert once until problem clears

򐂰 Option 3: Only generate alerts every x minute(s)/hour(s)/day(s)

With option 2, no notification is sent while the problem persists. Therefore, there is no
difference between (2) and (4) in the number of notifications that are sent. To get notified if
the problem still persists, select this option.
Figure 3-29 shows that while the threshold violation persists, notifications are sent out
every x minute(s)/hour(s)/day(s) (5).

5 5 5 5 5 5 5

Figure 3-29 Only generate alerts every x minute(s)/hour(s)/day(s)

Chapter 3. Setting thresholds and alerts 93

 򐂰 Option 4: Do not alert until condition has been violated for more than x minute(s)/hour(s)/
day(s)
Sometimes it is not necessary to be notified if a threshold gets violated just a few times
(6). In this case, option 4 might be a good choice. Notifications will only be sent out if the
threshold is been violated for more than x minute(s)/hour(s)/day(s) (7), as shown in
Figure 3-30.

6 7

6 6
Figure 3-30 Do not alert until condition has been violated for more than x minute(s)/hour(s)/day(s)

Note: Additional information about how to set up alerts and thresholds is available in IBM
Spectrum Family: IBM Spectrum Control Standard Edition, SG24-8321, and in the IBM
Spectrum Control section of IBM Knowledge Center.

3.3 Guidelines for thresholds

The combination of alerts with thresholds in IBM Spectrum Control enables you to automate
alerts and customize them for your environment. This automation can save you precious time
by avoiding manual checks, and allows faster reaction to events because alerts are sent
immediately after being triggered. The administrator can take appropriate actions as soon as
the warning or alert is received. It is also possible to run a script based on such alerts for
standardized events, completely eliminating any need for manual intervention.

However, do not underestimate the task of preparing and defining the required thresholds for
your environment. The better the thresholds and alerts are set, the better the whole system
works and the less manual intervention is needed here. Every system, every environment,
and every setup has its unique characteristics, so there is no generic rule and predetermined
values for the various metrics to monitor.

It is critical that every parameter be set based on your specific setup and needs. Also, your
environment is probably not static and your settings will need to be reevaluated and adjusted
over time. Changes to your environment might be required by an alternation of the existing
system, such as a change of the SAN speed, or the introduction of a new storage system.

This section provides guidance on how to select the right setting for specific aspects of your
environment.

94 Managing FlashSystem Performance with IBM Spectrum Control

3.3.1 Service level agreements
A direct source of information are the internal service level agreements (SLA) mandated in
your environment. SLAs are your first source to set thresholds on the corresponding system.
If there is an SLA in place that says application “gold” must have latencies better then x ms,
then set the alerts close to that value, but with some tolerance. You can do so for every
parameter specified in an SLA.

3.3.2 Planning and sizings

To avoid overloading a complete system, use the planning and sizing information to help you
determine an upper limit for specific characteristics of your environment and set thresholds
accordingly. For example, if a system is able to sustain a maximum 100,000 IOPS, set the
threshold to trigger an alert at a value slightly lower than that maximum to help keep the
system workload in the range that it was sized for.

Note: There are some metrics in Spectrum Control for which an alert cannot be set. Check
in advance if a particular alert can be set or if a combination of other values can act as a
workaround to reach the same goal.

3.3.3 Work with your own historical data

IBM Spectrum Control allows you to easily get a summary of the history of many metrics.
That history can help you determine the appropriate value to set as the threshold for
triggering alerts on those metrics, as observed in your environment. We illustrate in a brief
example, based on FlashSystem 840, an approach that you can use for many other metrics.

Chapter 3. Setting thresholds and alerts 95

 In IBM Spectrum Control, open the Alerts Definition window for the system in scope. In our
example, the Ports tab is selected with Performance metrics, as shown in Figure 3-31.

Selected part and metric

Icon to open the detailed view

for this metric

Figure 3-31 IBM Spectrum Control Alerts definition tab

For the resource and metrics that you selected, you can view the typical range in the short
summary and set the threshold based on that range. The advantage here is that the user is
already on the Alerts tab and knows which thresholds can be set. If you want to see a more
detailed history and have more options, you can use the normal performance view of IBM
Spectrum Control.

96 Managing FlashSystem Performance with IBM Spectrum Control

Click the Chart icon next to the metric (Port Receive Bandwidth Percentage in our example)
to open a chart, as shown in Figure 3-32.

Predefined periods

Alert
Warning

Customize shown
timeframe

Scale to change the

warning trigger level

Figure 3-32 Alert Definition detailed view

In this view, the history of this single metric is shown. The time frame can be changed by
selecting any of the predefined periods, such as 1 hour, 1 day, 1 week, and so on. You can
also customize the time frame.

The yellow line around the 75% in the chart indicates the trigger for a warning, whereas the
red one at around 85% is the trigger for an alert.

In the bottom part of the window, you can adjust the value for the warning threshold, which
repositions the yellow line.

In this example, you can see that the actual values measured (green curve) are far away from
the preset warning and alert thresholds. If wanted, you can adjust them to a closer level.

Make sure of the following before deciding on what value to use:

򐂰 Use at minimum a one week time frame to cover at least every weekday, including the
weekend. If you have special workloads, for example at end of the month, include this time
frame to cover those particular workload situations.
򐂰 Evaluate whether the alert threshold must be below the real chart (for example, cache hit
ratio) line or above (for example, CPU usage).
򐂰 Evaluate whether this threshold is to secure an SLA with a client, or is it just to get a better
knowledge of your system or to prepare future planning.

Chapter 3. Setting thresholds and alerts 97

 The history of metrics can also be viewed in IBM Spectrum Control standard charts with more
details than what you get in the Alerts windows shown in Figure 3-31 on page 96 and
Figure 3-32 on page 97. The standard charts provide a more granular view for single
resources like a single port.

To better define the settings for thresholds and alerts, you can also combine different values.
For example, when looking at latency on ports, you can also set as a condition (filter) that
there must be a minimum IOPS on the same port. If there is only a small amount of IOPS on
a single resource like a port, a high latency alert might not be relevant. To filter them out and
avoid false alarms, set a minimum of IOPS condition along with the latency alert. Another
example would be to combine some metrics for volumes with a minimum of space usage. Or
you could also combine bandwidth usage in relation to I/O block size.

In summary, whether you rely on SLAs or historical values, or a combination of both,

Spectrum Control gives you a lot of flexibility on how to specify alerts and thresholds.
Remember to revalidate your settings whenever changes occur in your environment or based
on your observations over time.

3.3.4 Device-specific metrics

As already stated, every environment has its own characteristics and properties, so it is
impossible to recommend values that will fit every situation. To give some initial guidance, we
included some examples, and when applicable, suggested specific values.

Important: Keep in mind that we are discussing examples. Before adopting any of the
suggested settings, carefully review the characteristics of your environment. These
examples are meant to illustrate the type of thinking that must take place when you decide
on which metrics and thresholds to use. They are by no means an exhaustive list of the
situations that need to be considered in your environment.

IBM FlashSystem 840/900

This section covers several important metrics for which it is useful to set warnings and alerts.
This is not a complete list and these might not fit every case.

Data Rate (MiB/s) of ports

It is important to monitor what amount of data is transferred through the ports. Check that the
physical bandwidth limitation is not reached. It is usually worth to combine the amount with
transfer size because numerous small IOs can saturate a port faster than the bandwidth
shows.

Response time (ms/op) of ports

The response time provides the average number of milliseconds that it took handle a transfer
(send or receive). This time is measured for every port, but the alert is for all or at least a
group of ports. Keep this fact in mind when you set the threshold. Depending on your
situation, it might be worthwhile to use the general groups function to separate them.

If the response is too high, try to find the reason and fix the problem, or balance the ports if it
is caused by excessive load and uneven use of ports.

Flash Health percentage of Flash modules

This is not a performance-related measurement. However, decreasing health can lead to
decreased performance, and should be checked as a possible reason.

98 Managing FlashSystem Performance with IBM Spectrum Control

IBM FlashSystem V9000
The V9000 offers many features such as compression and Remote Mirroring. All these
functions have their own metrics and should be monitored as well. This section focuses on
the general performance functions.

CPU Utilization (%)

The CPU utilization is good first indicator of the overall usage of the system. The overall
utilization is the primary threshold that should be set. Because one control enclosure must
occasionally handle the load of the second enclosure when it goes offline (such as due to
software upgrade, power failure, and so on), the CPU utilization threshold should not be set
too high. A good value to start with is 60% to trigger an alert.

If this limit is reached regularly, it can be an indicator that you need to distribute the load to
other control enclosures or to plan an upgrade with more or newer hardware.

Managed Disks response time (ms/op)

Because there is no usable measurement of the internal drives, the managed disks metrics
are an acceptable alternative for setting an alert. A response time issue with managed disks
might indicate that the internal disks are themselves experiencing a degraded response time.

Note: The internal Flash modules or drives have much better response times than an
external NL-SAS device. Keep this fact in mind when setting alerts thresholds.

The best parameter on which to set a threshold is the response time for the MDisks. Keep in
mind that different types of backend storage have different characteristics, so a grouping can
be helpful here as well.

Port metrics
The port metrics on a V9000.are a very important indicator. The Fibre Channel ports are the
interface for many types of workload. This is the interface for communication within the
system and with its internal and back-end storage, and also, in most cases, the interface to
hosts.

The IBM FlashSystem V9000 offers a huge variety of SAN-related counters that allow you to
proactively monitor the environment.

Data Rate (MiB/s) of ports

This metric indicates data transferred through the ports. Check that the physical bandwidth
limitation is not reached, and set the alerts lower than the physical maximum. It is valuable to
combine the data rate with transfer size because IO with a small transfer size can saturate a
port faster than the bandwidth indicates.

Response time (ms/op) of ports

The response gives the average number of milliseconds that it took handle a transfer (send or
receive). This time is measured for every port, but the alert is for all or at least a group of
ports. Have this consideration in mind when you set the threshold. Depending on your setup,
it might be valuable to use the general groups function to separate them.

On the V9000, it is also important to separate the type of load. The type of load can be
monitored at a node level. The node to local node traffic should see lower latencies than
traffic to host remote node. With local node to node traffic such as cache mirroring, it is
important to have low latencies, while a remote node at long distance will experience higher
latencies. A granular setting of alerts is important here.

Chapter 3. Setting thresholds and alerts 99

 Error rates on ports
In general, any error in the SAN is not good, but some are unavoidable. If you monitor error
rates, set the alert threshold at an acceptable but nonzero value to avoid flooding your
mailbox with alerts.

The value with highest focus and lowest alert trigger level should be the Cyclic Redundancy
Check (CRC) error rate. The CRC error rate should be set low. When it starts increasing, an
immediate reaction is needed.

In addition, the other error rates like Link Failures, Signal Loss, and Sync Loss are signs of a
broken connection. However, note that such errors can happen during a server reboot or
other temporary events and not be the indication of a problem. Alternately, higher and steadily
increasing error rates values can be a sign of a problem.

Another port metric, which might not be included in other SAN monitoring tools, is the Zero
Buffer Credit metric. IBM Spectrum Control allows you to monitor the Zero Buffer Credit Timer
(µs) or the Zero Buffer Credit Percentage (%). While the timer shows the number of
microseconds for which the port has been unable to send frames due to the lack of buffer
credit since the last node reset, the percentage gives the value in relation to the total load.
The percentage is typically the value that you want to monitor. Both metrics are from the
FlashSystem V9000 perspective, and thus indicate that the FlashSystem V9000 is not able to
send data out to the SAN.

Important: Note that the Zero Buffer Credit metrics are not reported with 26 Gb HBAs.
However, starting with VSC code V7.8.1, alternative metrics were introduced: Port Send
Delay Time and Port Send Delay I/O Percentage.

Figure 3-33 shows the view in IBM Spectrum Control.

Figure 3-33 New metrics for Port Send delay to replace Zero Buffer Credit

The Port Send Delay Time and Port Send Delay I/O Percentage can be used instead of the
Zero Buffer Credit, including for 8 Gb ports. If there are two types in use in the same system,
only one metric is enough to monitor both.

100 Managing FlashSystem Performance with IBM Spectrum Control

 Volume compression ratio
Because a good practice for enabling compression on a volume is when you can get a
minimum of 40% compression ratio, it is worthwhile to set an alert to be triggered when that
40% threshold is not met. However, a nearly empty volume, like a newly created one, can
show bad ratios initially. Therefore, it is relevant to combine the compression ratio and the
used capacity. In our example in Figure 3-34, we set the minimum used capacity at 10%. This
setting means that the 40% compression threshold is only taken into account for creating an
alert if at least 10% of the volume are used.

Figure 3-34 Combine Compression Savings and Used Allocated Space

Chapter 3. Setting thresholds and alerts 101

102 Managing FlashSystem Performance with IBM Spectrum Control
 A

Appendix A. Using SSH key for device

configuration
For some storage systems, such as FS840, FS900, V840, and V9000, there are two options
when configuring a device to IBM Spectrum Control:
򐂰 Username and Password
򐂰 SSH Key:
– Upload new key
– Upload an existing SSH key

You can either use the keys that are provided with IBM Spectrum Control in
...\IBM\TPC\device\conf, or create your own keys using a key generator such as
PuTTYgen. Creating your own keys is safer and therefore preferred.

If you use multiple IBM Spectrum Control servers consider to generate SSH Key pairs for
each IBM Spectrum Control server, which will allow you to see which IBM Spectrum Control
server is performing which operation on a device eg. to disable /enable access for each IBM
Spectrum Control Server.

In this paragraph we will describe how to generate SSH keys using PuTTYgen and how to
configure a FS840 to IBM Spectrum Control in using the previously created SSH keys.

© Copyright IBM Corp. 2018. All rights reserved. 103

Configuring a device using an SSH key
Before generating key pairs, the number of bits in a generated key can be adjusted (1), then
the key pair generation can be started by using the Generate key pair function (2), as
shown in Figure A-1.

Figure A-1 Generate key pair

104 Managing FlashSystem Performance with IBM Spectrum Control

While the progress bar proceeds, move the mouse over the blank area (1), as shown in
Figure A-2.

Figure A-2 Key generation

Appendix A. Using SSH key for device configuration 105

 After the keys have been created, save the public key as .pub and the private key as .ppk (1).
A passphrase (2) can be specified to increase security, as shown in Figure A-3.

Figure A-3 Save keys

106 Managing FlashSystem Performance with IBM Spectrum Control

If the storage system does not accept the .ppk key, convert it to an OpenSSH (.pem) key, as
shown in Figure A-4.

Figure A-4 Convert private key to OpenSSH key

To upload the public key to the storage system create a user on the storage system and
upload the public key, as shown in Figure A-5.

Figure A-5 Upload public key to storage system

Appendix A. Using SSH key for device configuration 107

 To configure the storage device to IBM Spectrum Control, change the Authentication
mechanism to Secure Shell, select Use an existing SSH key, and upload the appropriate
private key, as shown in Figure A-6.

Figure A-6 Configure FS840 with SSH key

108 Managing FlashSystem Performance with IBM Spectrum Control

Related publications

The publications listed in this section are considered particularly suitable for a more detailed
discussion of the topics covered in this paper.

IBM Redbooks
The following IBM Redbooks publications provide additional information about the topic in this
document. Note that some publications referenced in this list might be available in softcopy
only.
򐂰 IBM FlashSystem A9000 and IBM FlashSystem A9000R Architecture and
Implementation, SG24-8345
򐂰 IBM Spectrum Family: IBM Spectrum Control Standard Edition, SG24-8321
򐂰 IBM Tivoli Storage Productivity Center V5.1 Technical Guide, SG24-8053
򐂰 IBM Tivoli Storage Productivity Center V5.2 Release Guide, SG24-8204
򐂰 Implementing IBM FlashSystem 900 Model AE3, SG24-8414
򐂰 Implementing IBM FlashSystem V9000 - AC3 with Flash Enclosure Model AE3,
SG24-8413
򐂰 Reporting with TPCTOOL, REDP-4230

You can search for, view, download or order these documents and other Redbooks,
Redpapers, Web Docs, draft and additional materials, at the following website:
ibm.com/redbooks

Online resources
These websites are also relevant as further information sources:
򐂰 IBM Spectrum Control section of IBM Knowledge Center
https://www.ibm.com/support/knowledgecenter/SS5R93

Help from IBM

IBM Support and downloads
ibm.com/support

IBM Global Services

ibm.com/services

© Copyright IBM Corp. 2018. All rights reserved. 109

110 Managing FlashSystem Performance with IBM Spectrum Control
Back cover

REDP-5381-00

ISBN 0738456810

Printed in U.S.A.

®
ibm.com/redbooks