ABC of Teradata System Performance

Analysis

Shaheryar Iqbal
[email protected]

GCC Pakistan.
Date created: Feb 20, 2009.
Last Updated: Oct 12, 2009.
Agenda

1) Co-existing systems parallel efficiency

2) CPU Reports
1) CPU Utilization
2) CPU Node Hours available
3) Percent Shift Busy
4) OS % of CPU
5) CPU Utilization among Vprocs
6) Nodes Parallel Efficiency
7) AMPs Parallel Efficiency

3) Memory Reports
1) Memory availbilty
2) Mem allocation Failures

4) Disk I/O Reports

1) Disk busy %
2) I/O Wait
3) Measuring I/O
4) Disk Read Writes
5) Full cylinder Reads
6) Logical vs. Physical Reads
7) Mini Cyl Packs

5) Buddy Backup Effectiveness

6) AWT Usage

7) Host Utilities Report

8) Charting Teradata Manager Reports

2 > 1 August 2024
Feed back

______________________________________________
From: Clark, Dave
Sent: Saturday, March 07, 2009 12:14 AM
To: Iqbal, Shaheryar
Subject: ABC of Teradata System Performance Analysis

Shaheryar-

I have approved your presentation for general viewing. Thank-you very much for your
effort in developing this. The information is very useful.

-dave.clark
(858)485-2177

• http://pc01.teradata.com/CKS

3 > 1 August 2024

1) Co-existing systems parallel efficiency

4 > 1 August 2024

Co-existing systems parallel efficiency
Calculation & Reconfiguration

Gain in Co-exiting systems parallel

efficiency is the source of the
biggest gain in performance.

After Reconfiguration:
Before Reconfiguration:
Unusable Node Capacity
Unusable Node Capacity
14.23 -14.20 = 0.03;
14.23 – 13.25 = 0.98;
Parallel Efficiency = 99.80
Parallel Efficiency = 93.09
5 > 1 August 2024
• 2) CPU Reports

6 > 1 August 2024

CPU Utilization Chart Higa Macro
ResPmaTotal

Generated with Higa Macro ResPMATotal

7 > 1 August 2024
 Absolute CPU Node Hours available Higa Macro
ResPmaThreeShifts

•ABS Avail Hours: Time when CPU was available to process a job but there was nothing to process
•This graph gives a ratio between total CPU available to Max CPU used on daily basis
8 > 1 August 2024
 CPU Utilization: Percent Shift Busy Higa Macro
ResPmaThreeShifts

Shift Time 00-07 Shift Time 07-17 Shift Time 17-24 Shift Time 00-24
Mon-Fri Mon-Fri Mon-Fri Sat-Sun

•From previous slide same graph shown in Percent shift busy

9 > 1 August 2024

OS % of CPU Higa Macro
ResPmasec

In-Efficient Use of CPU

•CPU 100 % Busy

•OS% of CPU less than 20%
•For more than 30 minutes

Extremely degraded
system performance

10 > 1 August 2024

CPU Utilization among Vprocs Example #1 Higa Macro
ResSvprCPUs

A normal breakdown of the total CPU

utilization among the AMP, PE and node vprocs

11 > 1 August 2024

CPU Utilization among Vprocs Example #2 Higa Macro
ResSvprCPUs

A very high CPU usage by PE. (Non

optimized ) TPUMP sessions one of
the reasons

PE consuming 40% of
Node CPU, works at
their max capacity.
And have tendency to
become bottleneck.

12 > 1 August 2024

 Nodes Parallel Efficiency – Example 1 Higa Macro
ResPmaTotal

Extreme case
of Node skew

•Node Skew = Max Node CPU – Avg Node CPU

13 > 1 August 2024
 Nodes Parallel Efficiency – Example 2 Higa Macro
ResPmaTotal

High Parallel Efficiency

among Nodes

Node Skew = Max Node CPU – Avg Node CPU

14 > 1 August 2024
AMPs Parallel Efficiency – Example 1 Higa Macro
ResSvprCPUs

High parallel Efficiency is present among AMPs

15 > 1 August 2024

AMPs Parallel Efficiency – Example 2 Higa Macro
ResSvprCPUs

Parallel Efficiency not good among AMPs at times

16 > 1 August 2024

AMPs Parallel Efficiency – Example 3 Higa Macro
ResSvprCPUs

Extreme case of skew among AMPs

17 > 1 August 2024

• 3) Memory Reports

18 > 1 August 2024

 0
200
400
600
800
1 000
1 200
12/18 Sat 00:10

12/20 Mon 23:10

12/23 Thu 21:50

12/26 Sun 20:30

12/29 Wed 19:10

01/01 Sat 17:50

01/04 Tue 16:30

01/07 Fri 15:10

01/10 Mon 13:50

Coexistence View

01/13 Thu 12:30

01/16 Sun 11:10

Memory Utilization

12/19 Sun 17:20

12/22 Wed 16:00

12/25 Sat 14:40

12/28 Tue 13:20

12/31 Fri 12:00

01/03 Mon 10:40

01/06 Thu 09:20

01/09 Sun 08:00

01/12 Wed 06:40

19 > 1 August 2024

• Can FSGcache adjustment help?
01/15 Sat 05:20

12/18 Sat 11:10

12/21 Tue 10:10

12/24 Fri 08:50

12/27 Mon 07:30

Average & Minimum Free Memory Available

12/30 Thu 06:10

01/02 Sun 04:50

01/05 Wed 03:30

01/08 Sat 02:10

01/11 Tue 00:50

01/13 Thu 23:30

01/16 Sun 22:10

is experiencing
low memory condition
ResPma

One of the node groups

Higa Macro

Mem Free ------

Mem Free ------
 Memory Utilization Higa Macro
ResPma

There are low memory or depletions present, as marked with

circles. But there frequency of occurrence is not alarming.

•Available Free memory less than 100 MB and 40 MB are termed as “Memory depletion” and “system panic” states respectively.

20 > 1 August 2024

Paging & Memory Allocation Fails Higa Macro
ResPma

Mem Alloc Fails occurs when FREE MEM reaches to zero.

Negligible Mem Alloc Failures

occurred in last month.

21 > 1 August 2024

• 4) Disk IO Reports

22 > 1 August 2024

DISK IO: Disk % Busy Higa Macro
ResSldvNode

System is not IO bound: Disk doesn't remain 100% busy.

•Disk Busy time is the amount of time in which there is at least one I/O request outstanding

23 > 1 August 2024

 0
20
40
60
80
100
120
03/01 Wed 00:10

03/01 Wed 09:50

03/01 Wed 19:30

03/02 Thu 05:10

03/02 Thu 14:50

03/03 Fri 00:30

03/03 Fri 10:10

03/03 Fri 19:50

target range
03/04 Sat 05:30

Ratio is well within

03/04 Sat 15:10

03/05 Sun 00:50

03/05 Sun 10:30

03/05 Sun 20:10

03/06 Mon 05:50

03/06 Mon 15:30

I/O Wait% Example #1

• Average CPU + I/O Wait = System Busy

03/07 Tue 01:10

03/07 Tue 10:50

03/07 Tue 20:30

03/08 Wed 06:10

Chart Type: Stacked Area

24 > 1 August 2024

03/08 Wed 15:50

03/09 Thu 01:30

03/09 Thu 11:10

Average Cpu Busy vs. I/O Wait

03/09 Thu 20:50

03/10 Fri 06:30

03/10 Fri 16:10

03/11 Sat 01:50

03/11 Sat 11:50

03/11 Sat 21:30

03/12 Sun 07:10

03/12 Sun 16:50

03/13 Mon 02:30

• If Avg CPU + I/O Wait = 100%, desirable ratio is >= 90% avg CPU to >= 10% I/O Wait
Higa Macro
ResPmaTotal

Wait % ----
CPU bsy ---
 0
20
40
60
80
100
120
1/9 Fri 0:00

1/9 Fri 8:50

1/9 Fri 17:40

1/10 Sat 2:30

1/10 Sat 11:20

1/10 Sat 20:10

1/11 Sun 6:20

1/11 Sun 15:10

1/12 Mon 0:00

1/12 Mon 8:50

1/12 Mon 17:40

1/13 Tue 2:30

1/13 Tue 11:20

1/13 Tue 20:10

1/14 Wed 5:00

1/14 Wed 13:50

1/14 Wed 22:40

1/15 Thu 7:30
I/O Wait% Example #2

1/15 Thu 16:20

1/16 Fri 1:10

1/16 Fri 10:00

1/16 Fri 18:50

1/17 Sat 3:40

1/17 Sat 12:30

Chart Type: Stacked Area

25 > 1 August 2024

1/17 Sat 21:20

1/18 Sun 6:10

Average Cpu Busy vs. I/O Wait

1/18 Sun 15:00

1/19 Mon 0:10
1/19 Mon 12:30

1/19 Mon 21:30

optimally balanced for power and throughput 1/20 Tue 6:20

1/20 Tue 15:10

1/21 Wed 0:00

• Wait I/O should be 10% or less for configuration

1/21 Wed 8:50

1/21 Wed 17:40

1/22 Thu 2:30

1/22 Thu 11:20

1/22 Thu 20:10

1/23 Fri 5:00

1/23 Fri 13:50

#1 40%-60% Wait I/O shown here
Higa Macro
ResPmaTotal

I/O Wait %
Avg CPU bsy
 50 000
100 000
150 000
200 000
250 000
300 000

0
06/15 Thu 06:20
06/15 Thu 06:20
06/15 Thu 06:20
06/15 Thu 06:20
06/15 Thu 06:20
Measuring I/O

06/15 Thu 06:30

06/15 Thu 06:30

were viewable with disk read/writes

06/15 Thu 06:30

On the system, differences in throughput

06/15 Thu 06:30
06/15 Thu 06:30

06/15 Thu 06:40

Chart Type: Stacked Area

26 > 1 August 2024

06/15 Thu 06:40
06/15 Thu 06:40
06/15 Thu 06:40
Disk Read KByte and Write KByte

06/15 Thu 06:40

06/15 Thu 06:50

06/15 Thu 06:50
06/15 Thu 06:50
06/15 Thu 06:50
06/15 Thu 06:50
Actual rated bandwidth for the configuration
Higa Macro
ResPmabyNode

Disk RdKB /Sec

Disk WrKB /Sec
Total Disk Reads/Writes Higa Macro
ResPmaTotal

Disk Position Reads, Pre-Reads and Writes

12 000 000

10 000 000

8 000 000

Total DB Wrts
6 000 000 Total Pre Rds
Total Position Rds

4 000 000

2 000 000

0
09/05 Mon 02:20

09/12 Mon 10:30

09/19 Mon 20:20

09/26 Mon 04:10

09/04 Sun 01:10

09/11 Sun 09:20

09/18 Sun 19:10

09/25 Sun 03:00

10/02 Sun 11:10

09/06 Tue 03:30

09/08 Thu 05:50

09/13 Tue 13:00

09/15 Thu 15:20

09/20 Tue 21:30

09/27 Tue 05:20

09/29 Thu 07:40

09/03 Sat 00:00

09/07 Wed 04:40

09/10 Sat 08:10

09/14 Wed 14:10

09/17 Sat 18:00

09/21 Wed 23:30

09/24 Sat 01:50

09/28 Wed 06:30

10/01 Sat 10:00

09/09 Fri 07:00

09/16 Fri 16:50

09/23 Fri 00:40

09/30 Fri 08:50

Total Pre-Reads are large proportion of Next step is to look at FCR denied cache.
total reads, so this system may If this is high, and there is wait I/O often,
be a good candidate for raisingChart Type: Stacked Area then this system is a candidate for
Full Cylinder Read slots higher FCR slots.

27 > 1 August 2024

Full Cylinder Reads Example #1 Higa Macro
ResFullCylReadTotal

Less FCR requests

with high
successful rate

28 > 1 August 2024

Full Cylinder Reads Example #2 Higa Macro
ResFullCylReadTotal

High FCR requests

with moderate
successful rate

29 > 1 August 2024

Logical vs. Physical Reads Higa Macro
SvprReadTotal

The more that the logical

reads exceeds physical
reads, the better use of
memory as a cache

30 > 1 August 2024

Mini-cyl-packs Higa Macro
ResCylPackTotal

Mini Cyl Pack occurs when there are only ten Free cylinders left on
any AMP resulting in degraded system performance.

31 > 1 August 2024

5) AWT Usage Higa Macro
ResSvprQLenAvg
ByVproc

Potential indicator of
“FLOW Control” state

Message queue
length > 20

32 > 1 August 2024

6) Buddy Backup Effectiveness Higa Macro
ResPmaBkupHour
Total

When there are many complete and few

partial blocks, sent to the buddy, then
the buddy backup should be turned off

33 > 1 August 2024

7) Host Utilities Traffic Higa Macro
reshostTotalHour

how much data is read

or written by utility

Total read/write traffic

per Hour, physical
MBytes transferred
between Host and Node.
34 > 1 August 2024
• 8) Charting Teradata Manager Reports

Use Teradata Manager Generate Custom

to record Active reports using pivot
Session detail data tables

35 > 1 August 2024

Charting Teradata Manager Reports:
CPU utilized per User

CPU utilized per User

36 > 1 August 2024

Charting Teradata Manager Reports:
CPU utilization Per Group

CPU utilized per

Performance Group

37 > 1 August 2024

Questions

The only bad question

is the question
never asked
38 > 1 August 2024