The Association of System

Performance Professionals

The Computer Measurement Group, commonly called CMG, is a not for profit, worldwide organization of data processing professionals committed to the
measurement and management of computer systems. CMG members are primarily concerned with performance evaluation of existing systems to maximize
performance (eg. response time, throughput, etc.) and with capacity management where planned enhancements to existing systems or the design of new
systems are evaluated to find the necessary resources required to provide adequate performance at a reasonable cost.

This paper was originally published in the Proceedings of the Computer Measurement Group’s 1992 International Conference.

For more information on CMG please visit http://www.cmg.org

Copyright Notice and License

Copyright 1992 by The Computer Measurement Group, Inc. All Rights Reserved. Published by The Computer Measurement Group, Inc. (CMG), a non-profit
Illinois membership corporation. Permission to reprint in whole or in any part may be granted for educational and scientific purposes upon written application to
the Editor, CMG Headquarters, 151 Fries Mill Road, Suite 104, Turnersville , NJ 08012.

BY DOWNLOADING THIS PUBLICATION, YOU ACKNOWLEDGE THAT YOU HAVE READ, UNDERSTOOD AND AGREE TO BE BOUND BY THE
FOLLOWING TERMS AND CONDITIONS:

License: CMG hereby grants you a nonexclusive, nontransferable right to download this publication from the CMG Web site for personal use on a single
computer owned, leased or otherwise controlled by you. In the event that the computer becomes dysfunctional, such that you are unable to access the
publication, you may transfer the publication to another single computer, provided that it is removed from the computer from which it is transferred and its use
on the replacement computer otherwise complies with the terms of this Copyright Notice and License.

Concurrent use on two or more computers or on a network is not allowed.

Copyright: No part of this publication or electronic file may be reproduced or transmitted in any form to anyone else, including transmittal by e-mail, by file
transfer protocol (FTP), or by being made part of a network-accessible system, without the prior written permission of CMG. You may not merge, adapt,
translate, modify, rent, lease, sell, sublicense, assign or otherwise transfer the publication, or remove any proprietary notice or label appearing on the
publication.

Disclaimer; Limitation of Liability: The ideas and concepts set forth in this publication are solely those of the respective authors, and not of CMG, and CMG
does not endorse, approve, guarantee or otherwise certify any such ideas or concepts in any application or usage. CMG assumes no responsibility or liability
in connection with the use or misuse of the publication or electronic file. CMG makes no warranty or representation that the electronic file will be free from
errors, viruses, worms or other elements or codes that manifest contaminating or destructive properties, and it expressly disclaims liability arising from such
errors, elements or codes.

General: CMG reserves the right to terminate this Agreement immediately upon discovery of violation of any of its terms.
 Learn the basics and latest aspects of IT Service Management at CMG's Annual Conference - www.cmg.org/conference

CAPACIlY MANAGEMENT IN A WIDELY DISTRIBUTED AS/400 NE1WORK

Randal J. Miller
Vincent Moy

Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
Waste Management, Inc.

Abstract
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe

For AS/400s in a widely distributed network, several challenges are presented in collectin~ transmittin~ and
evaluating peifonnance data that are generally not found in centralized MJIS computing environments. In a
networked AS/400 environmen4 the relative immaturity ofutilities and constraints oftelecommunications establish
limits to the volume of data that can be transferred. T1lerefore, decisions must be made to trade completeness of
data vermr the cost of transmitting the data back to the mainframe.

This paper will present an overview of OS/4oo V:U peifonnance data and address the capacity management
methodalogy adapted by Waste Managemellt to deal with the acquisitiOJI ofAS/400 peifomlance data from over
200 sites cOIIIIected in a widely distributed network.

INTRODUCTION BACKGROUND

One of the proofs of the success of the AS1400 since its Waste Management, Inc. is the world's leading
June, 1988 announcement has been the number of environmental services company. Based in Oak Brook,
companies that have attached distributed AS/400s to their Illinois, the company provides comprehensive solid and
mainframe MVS systems. However, this distributed bazardous waste services, energy, recovery and
environment raises some unique problems for capacity environmental technologies and engineering through
analysts when developing a capacity management subsidiaries operating throughout the United States,
methodology. From the mainframe perspective, capacity Canada, Europe, South America, the Middle East, and the
management usually involves the use of well established Pacific Region.
products, a limited number of systems, high speed
channels for data movement, and a vast disk/tape storage In order to enhance the support of North America field
pool to hold the performance data. However, capacity operations, tbe Customer Information Management
management in a distributed environment involves a System (CIMS) application was developed which was
limited product tool selection, numerous systems, relatively based upon a distributed processing architecture between
low speed network links, and a finite amount of data the field System/:36 systems and the host MVS mainframe.
storage capacity. Additionally, 3270 emulation was used to allow for access
to the mainframe financial systems. In 1988, the
The intent of this paper is to define an initial methodology introduction of tbe AS/400 technology, which supported
for AS/400 capacity management in a widely distributed the System/36 environment, allowed for the migration to
network from a centralized viewpoint. The following this larger platform. The increased capacity offered by the
sections define the background of the project, the network AS/400s also allowed for additional applications to be
environment, the AS/400 performance data and toolset, developed and deployed.
the problems that are raised in a distributed environment,
and the solutions implemented to overcome those With the increased use of tbe AS/400 capacity through
problems. This methodology should only be considered a both the deployment of the new applications and by
guideline as eacb environment will have unique increased use of the existing ones, a requirement to
measurement requirements that may not be included in actively monitor the performance of the field AS/400s was
this paper. established. In order to satisfy this requirement, a project
was initiated to develop a methodology for gathering and
reporting field AS/400 performance on a regular basis.

571

Find a CMG regional meeting near you at www.cmg.org/regions

 Learn the basics and latest aspects of IT Service Management at CMG's Annual Conference - www.cmg.org/conference

AS/400 NE1WORK PERFORMANCE TOOLS

Waste Management of North America's (WMNA) AS/400 The IBM AS/400 Licensed Performance Tools Product
network consists of over 200 systems distributed (performance Tools/400) provides a comprehensive set of
throughout the United States and Canada, all which tools to analyze AS/400 performance along with a capacity

Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
currently operate under OS/ 400 Version 2 Release 1. planning feature. The functions provided are shown in
While a limited number of the AS/400s deployed are rack Figure 2.
mounted B45 and D45 units, the majority of the systems
are of the chassis variety: models B, C, and D, lOs through
25s. The systems are presently linked to a host 9021-720 PROBLEMS IN A DlSTRIBlrrED ENVIRONMENT
via SDLC point-to-point 9600 bps analog circuits through
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe

one of two 3745 FEPs.

PERFORMANCE MONITORING

AS/400 PERFORMANCE MONITOR AND TOOLS For WMNA, the process of monitoring over 200 AS/400s
creates unique problems in gathering and ana\yziDg
For AS/400 performance management, IBM offers the performance data. These problems can be classified in
Performance Tools/400 product which evaluates the terms of:
performancc data supplied by the AS/400 performance
monitor. 1. Logistics
2. Technology
AS/400 PERFORMANCE MONITOR 3. Analysis

Integrated within the OS/400 operating system is a means Logistics. With over 200 AS/400s in geographically
of collecting performance data via the STRPFRMON different areas of the United States, the process of
command. The AS/400 performance monitor was controlling and monitoring the scheduling of performance
intended to collect performance data that would later data collection requires the implementation of automated
serve as input into the IBM AS/400 Licensed Performance facilities. In a widely distributed network, the analyst is
Tools Product. When invoked, a series of in-memory often removed from the immediate vicinity of the remote
registers collect performance data at key points in the processors. Since technical assistance is generally not
operation of the AS/ 400 and will periodically write the immediately available at most sites, the collection facility
information out to disk. must be able to initialize and terminate the performance
monitor and prepare the data for transmission back to the
The options selected within the STRPFRMON command host without manual intervention.
can effect the amount of performance data collected.
Some of the user specified parameters to the Technology. In a networked AS/400 environment, the
STRPFRMON command that control the amount of data relative immaturity of utilities and constraints of
collected arel : telecommunications establish limits to the volume of data
that can be transferred. The majority of the SDLC lines in
1. The time, date, and duration of performance the WMNA network have a speed of 9600 bps. On this
data collection line must travel 3270 emulation traffic, batch me transfers,
2. The type of data to be collected and printed report requests from the host mainframe.
3. The type of work that can be collected Also, during the evenings and early morning hours large
file transfers occur.
The control over the performance monitor is quite
cxtcnsivc and use of the monitor has minimal CPU It is within a six hour window that performance data can
overhead on the system (less than 2% with trace enabled). be transmitted back to the host mainframe. In the
Performance data can be obtained for virtually all WMNA network environment, there is an effective
hardware components of the AS/400 along with some transfer rate of approximately 1 Mb of data an hour per
network service information. site. However, a performance run with trace enabled for
one hour can create between 6 to 14 megabytes of data
AS/400 PERFORMANCE DATA depending upon the workload. A sampling run for 10
hours can create data in the range of 2 to 8 Megabytes,
The performance monitor collects and stores data in again depending upon workload.
physical files within the QPFRDATA library.2 A
description of the physical files is shown in Figure 1.

572

Find a CMG regional meeting near you at www.cmg.org/regions

 Learn the basics and latest aspects of IT Service Management at CMG's Annual Conference - www.cmg.org/conference

Analysis. Using the midpoint of data volume collected for command, monitor its execution, and post process the
a 10 hour sampling run (5 Megabytes), the daily volume of performance data collected.
performance data generated by 200 sites would consume
1 Gigabyte of data. Processing this volume through the TRANSMISSION OF DATA
Performance Tools/4OO package would greatly exceed the

Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
capacity of the C20 system that is used for performance Since the overhead of running the performance monitor is
data evaluation. Even if the data could be processed daily, low, the minimum sampling time interval of 5 minutes was
the quantity of reports generated would overwhelm the chosen to mlDimize the quality of the data. However, as
poor capacity analyst charged with monitoring the it was indicated earlier, transmitting the entire
performance of the field AS/4OOs. performance data library would severely stress the capacity
of the network. Therefore, the methodology had to
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe

Because of these logistical, technological, and analytical include a facility to reduce the amount of data transmitted
reasons, it becomes obvious that the use of the back to the host.
Performance Tools/4OO package is not a viable option for
supporting a widely distributed AS/4OO network. Data Reduction. The frrst step taken in reducing the
Therefore, some alternative method combining data amount of data transmitted was to reduce the amount of
reduction along with automated analysis is required. time the performance monitor was active. After initial
Additionally, the need to maintain historical data for review of field activity, it was determined that the
performance trending analysis further requires a method collection of performance data need only to be done
for reducing the volume of data. during the hours of normal on-line operations (08:00 to
18:00 hours). Furthermore, the sites would be sampled
In summary, the issues in a widely distributed AS/400 only once a week with the day of sampling randomized
network are: from week to week.

1. how to automate the collection of Finally, a subset of the performance dataset, not the entire
performance data, dataset, would be transmitted back to the host for
2. how to transmit the data within the processing. The subset would consist of only those
window allotted, and elements that were determined to be required for service
3. how to handle the analysis of the data. performance analysis of each field AS/4OO. The subset is
created by the CL monitoring program during the post
processing step by passing the performance data files
SOLUTION through a collection of Query programs. After post
processing, the amount of data to be transmitted is
AUfOMATION reduced by approximately 80%. This extracted subset can
then be more easily moved through the network back to
In order to address the first issue, automation of the host mainframe.
performance data collection, IBM's Netview Distribution
Manager (Netview/DM) was selected. Other products Pedonnance File Subset. The relevant performance data
that may be used for automated scheduling on AS/400s fields identified are shown in Figure 33.
are Advance System Research's PILOT, CA's
MASTERMIND, or Help Systems ROBOT. However, PERFORMANCE REPORTING
Netview/DM allowed for centralized control of scheduling
plus has a primary function of file transfer, a feature which Since only a subset of the performance data file is
would be used to transmit the performance data back to transmitted back to the host, the standard IBM
the host mainframe. Thus, by combining the scheduling Performance Tools/4OO facility cannot be used for
feature of Netview/DM with a custom written AS/4OO performance reporting. Thus, a customized reporting
Control Language (CL) program, automation of facility based on the relevant data fields extracted during
performance data collection was achieved. the creation of the subset was required.

Scheduling of tasks by Netview/DM is performed by At the management level, the extracted data is
invoking "PLANS" which are composed of multiple summarized into the SITE SUMMARY REPORT which is
"PHASES". Each Phase is associated with one AS/4OO generated weekly. The SITE SUMMARY REPORT has
site and contains "FUNCTIONS" or the set of commands two components, the SYSTEMS REPORT and the
executed by the Phase. Through the use of the Function COMMUNICA TlONS REPORT. The component reports
"INITIATE", a CLIST is executed which invokes the are initially reviewed by the capacity analyst to determine
custom written CL program to execute the STRPFRMON whether performance problems exist on any of the

573

Find a CMG regional meeting near you at www.cmg.org/regions

 Learn the basics and latest aspects of IT Service Management at CMG's Annual Conference - www.cmg.org/conference

monitored systems and then forwarded to the WMNA retained in a historical data me. As time
support group management. progresses, enough data for each site will
be accumulated to make historical trend
After creation of the report, the detail data is initially reports feasible. With these reports,
retained online and later moved to offline storage. If business and/or growth trends will

Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
performance problems are indicated by the management enhance the ability to make proactive
reports, the detail extracted data can be reanalyzed for capacity changes.
additional reporting. If further performance analysis is
required to diagnose a unique problem, a full performance 2. Exception Reports. With the number of
run will be executed and the complete QPFRDATA field AS/4OO systems growing. visual
library brought back either by the network or tape to be inspection of the increasing volume of
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe

processed in the PERFORMANCE TOOLS/4OO product. reports is becoming an undesirable

method of tracking site performance. An
Summary Report. Examples of the weekly SITE alternative would be to create an
SUMMARY REPORT are shown by Figures 4 and 6. In exception report where ouly those sites
Figures 4 and 5, the SYSTEMS REPORT is displayed that exceed Rule-of-Thumb performance
along with the data elements used in its creation that were thresholds are reported. However,
presented earlier in Figure 3. In Figures 6 and 7, the implementation of this "SITE
COMMUNICATIONS REPORT is shown along with its EXCEPTION REPORT" will require the
variable definitions. In both Figures 5 and 7, IBM establishment of response time Service
suggested Rule-of-Thumb (ROT) levels are indicated if Level Objectives which is a separate
they exist. project already underway.

3. Capacity Planning • Integrated within

LIMITATIONS Performance Tools/4OO is the only
known dedicated AS/4OO modeling
During the design and implementation of the WMNA facility called MDLSYS. Unfortunately,
customized AS/4OO performance toolset, some notable since MDLSYS requires the full
limitations were discovered. For instance, the data field QPFRDATA library as input, usage is
within the performance monitor which identifies the limited for site AS/4OO sizing. However,
transaction type is not being set properly. Therefore, the in OS /400 V2.1.1., IBM is replacing
AS/4OO performance monitor cannot provide information MDLSYS with a new modeling facility
about: written by BGS. Thus, it may be
possible to modify the extracted snbset
1. 5394 remote controller performance or me to include those elements required by
response times, the new facility thereby enhancing the
2. PC Support Work station response times, capacity planning effort.
and
3. Display station pass through response SUMMARY
times.
Limiting factors such as lack of network capacity, shortness
Further, since the data field is not updated, the monitor of batch transmission window, size of performance data
cannot differentiate between 3270 emulation response fdes, and the large number of AS/4OO systems make it
times and local host processing transactions. impractical to use the Performance Tools/4OO facility as
the vehicle for widely distributed AS/4OO performance
Finally, for non-transactional bounded applications reporting. Therefore, a methodology combining reduction
(applications that are not event driven), little performance of performance monitoring intervals, random sampling of
information is created by the trace facility. systems, and creation of a performance data me subset
was developed to overcome the limitations of getting data
FUTURES back to the mainframe. Although this required that
customized reports be written, as the cost of creating the
Enhancements to the current methodology under subset was the inability to use the Performance Tools/4OO
development include: standard reports, the customized reports allow for easy
review of the over 200 WMNA field systems. Also, control
1. Historical Trend Rewrts • As was of the subset will enable future enhancements to make
mentioned earlier, the extracted subset is monitoring of a larger number of systems possible.

574

Find a CMG regional meeting near you at www.cmg.org/regions

 Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe

Figure 2
FIgure 1

575

Find a CMG regional meeting near you at www.cmg.org/regions

Learn the basics and latest aspects of IT Service Management at CMG's Annual Conference - www.cmg.org/conference

Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
 Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe

Figure 3

576

Find a CMG regional meeting near you at www.cmg.org/regions

Learn the basics and latest aspects of IT Service Management at CMG's Annual Conference - www.cmg.org/conference

Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
 Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe

Figure 4

577

Find a CMG regional meeting near you at www.cmg.org/regions

Learn the basics and latest aspects of IT Service Management at CMG's Annual Conference - www.cmg.org/conference

Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
 Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe

578

Find a CMG regional meeting near you at www.cmg.org/regions

Learn the basics and latest aspects of IT Service Management at CMG's Annual Conference - www.cmg.org/conference

Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
 Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe

Figure 7
Figure 6

579

Find a CMG regional meeting near you at www.cmg.org/regions

Learn the basics and latest aspects of IT Service Management at CMG's Annual Conference - www.cmg.org/conference

Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
 Learn the basics and latest aspects of IT Service Management at CMG's Annual Conference - www.cmg.org/conference

l.For further information on performance data collection, please refer to the AS/4OO Programming: Performance
TooIs/4OO Guide, Chapter 4; Collecting System Performance Data.

Buy the Latest Conference Proceedings and Find Latest Computer Performance Management 'How To' for All Platforms at www.cmg.org
2.A complete reference of each file may be obtained from the AS/4OO Programming: Work Management Guide,
Appendix A; pages A-9 through A-66.

3.A complete reference of each field may be obtained from the AS/4OO Programming: Work Management
Guide, Appendix A; pages A-9 through A-66.
Join over 14,000 peers - subscribe to free CMG publication, MeasureIT(tm), at www.cmg.org/subscribe

580

Find a CMG regional meeting near you at www.cmg.org/regions