Progress Database Performance Tuning
Progress Database Performance Tuning
WORKSHOP - ARCHITECTURE
Adam Backman
President and Pretty Nice Guy
White Star Software, LLC
Overview
• OpenEdge Architecture
– Shared memory
– Server-less
– Multi-server
• Networking
– Primary broker
• Splitting clients across servers
– Secondary broker
• Splitting clients across brokers
Overview
• Database block size
• Setting records per block
• Using OE Type II Storage areas
Overview
• Disk Stuff
• Use RAID 10
• Use large stipe widths
• Match OpenEdge and OS block size
Architecture
I think Ms. Monroe’s architecture is extremely
good architecture
-Frank Lloyd Wright
OpenEdge Memory Architecture
• Shared memory
• Server-less
• Multi-server
• Multi-broker
OpenEdge Memory Architecture
Shared Memory
Remote Broker
Client
Server
Background Processes
Database
7
OpenEdge Network Architecture
• Primary broker
• Splitting clients across servers
• Secondary broker
• Splitting clients across brokers
OpenEdge Architecture
Client/Server Overview
• The OpenEdge Server
– A process that accesses the database for 1 or
more remote clients
4GL
Server
Database
9
OpenEdge Storage Considerations
• Database block size
• Setting records per block
• Type II Storage areas
Database Block Size
• Generally, 8k works best for Unix/Linux
• 4k works best for Windows
• Remember to build filesystems with larger
block sizes (match if possible)
• There are exceptions so a little testing
goes a long way but if in doubt use the
above guidelines
Determining Records per Block
• Determine “Mean” record size
– Use proutil <dbname> -C dbanalys
• Add 20 bytes for record and block overhead
• Divide this product into your database block
size
• Choose the next HIGHER binary number
– Must be between 1 and 256
Example: Records /Block
Fixed extent
Cluster Database block
Variable extent
Not yet
allocated
by O/S
Filled Partly filled Free Not yet allocated
Type II Storage Areas
• Data is clustered together
• A cluster will only contain records from a single
table
• A cluster can contain 8, 64 or 512 blocks
• This helps performance as data scatter is
reduced
• Disk arrays have a feature called read-ahead
that really improves efficiency with type II areas.
Type II Clusters
Fixed Extent
Gus Björklund
>
> probkup foo
>
2
9
>
> proserve foo
>
“Out of the box” Performance
800
700
600
throughput (tps)
500
400
300
200
100 30 0
0
no tuning simple tuning
>
> proserve foo -B 32000
>
2: Spinlock retry count
>
> proserve foo -B 32000 -spin 5000
>
3: Start BI Log Writer (BIW)
>
> proserve foo -B 32000 -spin 5000
> probiw foo
>
4: Start Async Page Writer (APW)
>
> proserve foo -B 32000 -spin 5000
> probiw foo
> proapw foo
> proapw foo
>
5: Increase BI Log Block Size
>
> proutil foo -C truncate bi \
> -biblocksize 8
> proserve foo -B 32000 -spin 5000
> probiw foo
> proapw foo
> proapw foo
>
6: Increase BI Log Cluster Size
>
> proutil foo -C truncate bi \
> -biblocksize 8 -bi 4096
> proserve foo -B 32000 -spin 5000
> probiw foo
> proapw foo
> proapw foo
>
7: Add BI Log buffers
>
> proutil foo -C truncate bi \
> -biblocksize 8 -bi 4096
> proserve foo -B 32000 -spin 5000 \
> -bibufs 25
> probiw foo
> proapw foo
> proapw foo
>
8: Fix Database Disk Layout
here everything on same disk, maybe with other stuff
500
400
300
200
100 30
0
no tuning easy tuning
300 260
250
200
150
100
50
0
32,000 64,000 128,000 256,000
-B setting
Next, the lab,
but first:
Questions
Database Performance Tuning
Workshop
Big B
A Few Words about the Speaker
• Tom Bascom; free-range Progress coder &
roaming DBA since 1987
• VP, White Star Software, LLC
– Expert consulting services related to all aspects of
Progress and OpenEdge.
– [email protected]
• President, DBAppraise, LLC
– Remote database management service for OpenEdge.
– Simplifying the job of managing and monitoring the
world’s best business applications.
– [email protected]
51
What is a “Buffer”?
• A database “block” that is in memory.
• Buffers (blocks) come in several flavors:
– Type 1 Data Blocks
– Type 2 Data Blocks
– Index Blocks
– Master Blocks
Block Layout
Block’s DBKEY Type Chain Backup Ctr Block’s DBKEY Type Chain Backup Ctr
Next DBKEY in Chain Block Update Counter Next DBKEY in Chain Block Update Counter
Num Free
Dirs. Dirs. Free Space Rec 0 Offset Rec 1 Offset Top Bot Index No. Reserved
Rec 2 Offset Rec n Offset Num Entries Bytes Used Dummy Entry . . .
row 1
. . . Compressed Index Entries . . .
row 2
Free Space
row 0
2 1 19 2.75 Shipped
DKP Pitt, Dirk K. 1,800
2 2 49 6.78 Shipped
4 Go Fishing Ltd Harrow
2 3 13 10.99 Shipped
16 Thundering Surf Inc. Coffee City
55
Type 2 Storage Area
Block 1 Block 3
1 Lift Tours Burlington 9 Pihtiputaan Pyora Pihtipudas
56
What is a “Buffer Pool”?
• A Collection of Buffers in memory that are
managed together.
• A storage object (table, index or LOB) is
associated with exactly one buffer pool.
• Each buffer pool has its own control structures
which are protected by “latches”.
• Each buffer pool can have its own
management policies.
Why are Buffer Pools
Important?
58
Locality of Reference
• When data is referenced there is a high
probability that it will be referenced again
soon.
• If data is referenced there is a high probability
that “nearby” data will be referenced soon.
59
Which Cache is Best?
# of Cost per
Layer Time Recs # of Ops Op Relative
Progress 4GL to –B 0.96 100,000 203,473 0.000005 1
etime( yes ).
do while lastkey = -1 and etime < 20: /* pause 0.05 no-message. */
readkey pause 0.
end.
end.
return.
Isn’t “Hit Ratio” the Goal?
• No. The goal is to make money*.
95% 90.000
90.0%
20,000 80.000
OSRd
HR 70.000
Time
15,000 60.000
50.000
10,000 40.000
30.000
95% = plenty of room for improvement
5,000 20.000
10.000
0 0.000
5,000
75,000
100,000
125,000
150,000
200,000
300,000
400,000
500,000
600,000
700,000
800,000
900,000
1,000,000
Hit Ratio Summary
• If you must have a “rule of thumb” for HR:
• 90% terrible.
• 95% plenty of room for improvement.
• 98% “not bad”.
• The performance improvement from
improving HR comes from reducing disk IO.
• Thus, “Hit Ratio” is not the metric to tune.
• In order to reduce IO operations to one half
the current value –B needs to increase 4x.
Exercises
68
Exercise 0 - step 1
# . pro102b_env
# cd /home/pace
# proserve waste –B 3250000
# start0.0.sh
OpenEdge Release 10.2B03 as of Thu Dec 9 19:15:20 EST 2010
16:42:02 BROKER The startup of this database requires . . .
16:42:02 BROKER 0: Multi-user session begin. (333)
16:42:02 BROKER 0: Before Image Log Initialization . . .
16:42:02 BROKER 0: Login by root on /dev/pts/0. (452)
# pace.sh s2k0
...
Exercise 0 - step 2
┌──────────────────────────────────────┐
│Target Sessions: 10 │
│ │
│ Target Create: 50/s │
│ Target Read: 10,000/s │
│ Target Update: 75/s │
│ Target Delete: 25/s │
│ │
│ Q = Quit, leave running. │
│ │
│ X = Exit & shutdown. │
│ │
│ E = Exit to editor, leave running. │
│ │
│ R = Run Report workload. │
│ │
│ M = More, start more sessions. │
│ │
│ │
│ Option: __ │
└──────────────────────────────────────┘
Exercise 0 - step 3
• In a new window:
# . pro102b_env
# cd /home/pace
# protop s2k0
...
Exercise 0 - step 4
• Type “d”, then “b”, then <space>, then ^X:
Exercise 0 - step 5
Exercise 0 - step 6
• Type “d”, then “b”, then <space>, then “i”, then <space>, then “t”, arrow to “table statistics”, then <space> and finally ^X:
Exercise 0 - step 7
• On the “pace” menu, select “r”:
repOrder repLines repSales
otherOrder otherLines otherSales
──────────────────────────────────────────────────
20,436 247,478 $2,867,553,227.50
11,987 145,032 $1,689,360,843.35
Elapsed Time: 172.8 sec
-B: 102 -B2: 0
LRU: 47,940/s LRU2: 0/s
LRU Waits: 3/s LRU2 Waits: 0/s
-B Log IO: 47,928/s -B2 Log IO: 0/s
-B Disk IO: 3,835/s -B2 Disk IO: 0/s
-B Hit%: 92.00% -B2 Hit%: ?
My Log IO: 5,931/s
My Disk IO: 654/s
My Hit%: 88.97%
PUG Challenge USA
Performance Tuning
Workshop
Latching
Dan Foreman
Progress Expert, BravePoint
Introduction – Dan Foreman
• Progress User since 1984 (longer than Gus)
• Since Progress Version 2 (there was no
commercial V1)
• Presenter at a few Progress Conferences
Introduction – Dan Foreman
• Publications
– Progress Performance Tuning Guide
– Progress Database Administration Guide
– Progress Virtual System Tables
– Progress V10 DBA Jumpstart
Introduction – Dan Foreman
• Utilities
– ProMonitor – Database monitoring
– ProCheck – AppServer/WebSpeed monitoring
– Pro Dump&Load – Dump/load with minimum downtime
– Balanced Benchmark – Load testing tool
Apology
• Due to a flurry of chaos in my life the last few
weeks, I prepared this presentation while
riding an airport shuttle at 4am in the
morning….
Terminology
• Latch
• Latch Timeout (seen in promon)
• Spinlock Retries (-spin)
Server Components
• CPU – The fastest component
• Memory – a distant second
• Disk – an even more distant third
• Exceptions exist but this hierarchy is almost
always true
CPU
• Even with the advent of more sophisticated
multi-core CPUs, the basic principle of a
process being granted a number of execution
cycles scheduled by the operating system
Latches
• Exist to prevent multiple processes from
updating the same resource at the same time
• Similar in concept to a record lock
• Example: only one process at a time can
update the active output BI Buffer (it’s one
reason why only one BIW can be started)
Latches
• Latches are held for an extremely short
duration of time
• So activities that might take an indeterminate
amount of time (a disk I/O for example) are
not controlled with latches
-spin 0
• Default prior to V10 (AKA OE10)
• User 1 gets scheduled ‘into’ the CPU
• User 1 needs a latch
• User 2 is already holding that latch
• User 1 gets booted from the CPU into the Run
Queue (come back and try again later)
-spin <non-zero>
• User 1 gets scheduled into the CPU
• User 1 needs a latch
• User 2 is already holding that latch
• Instead of getting booted, User 1 goes into a
loop (i.e. spins) and keeps trying to acquire
the latch for up to –spin # of times
-spin <non-zero>
• Because User 2 only holds the latch for a short
time there is a chance that User 1 can acquire
the latch before running out of allotted CPU
time
• The cost of using spin is some CPU time is
wasted doing “empty work”
Latch Timeouts
• Promon R&D > Other > Performance
Indicators
• Perhaps a better label would be “Latch
Spinouts”
• Number of times that a process spun –spin #
of times but didn’t acquire the Latch
Latch Timeouts
• Doesn’t record if the CPU Quanta pre-empts
the spinning (isn’t that a cool word?)
Thread Quantum
• How long a thread (i.e. process) is allowed to
keep hold of the CPU if:
– It remains runnable
– The scheduler determines that no other thread
needs to run on that CPU instead
• Thread quanta are generally defined by some
number of clock ticks
How to Set Spin
• Old Folklore (10000 * # of CPUs)
• Ballpark (1000-50000)
• Benchmark
• The year of your birthday * 3.14159
Exercise
• Do a run with –spin 0
• Do another run with a non-zero value of spin
• Percentage of change?
PUG Challenge Americas
Performance Tuning Workshop
After Imaging
PAUL KOUFALIS
PRESIDENT
PROGRESSWIZ CONSULTING
Progresswiz Consulting
• Based in Montréal, Québec, Canada
• Providing technical consulting in Progress®,
UNIX, Windows, MFG/PRO and more
• Specialized in
– Security of Progress-based systems
– Performance tuning
– System availability
– Business continuity planning
Extents - Fixed versus variable
• Classic arguments:
– Better I/O to dedicated disks
– Can remove physical disks in case of crash
• Modern SANs negate both arguments
– My confrères may argue otherwise for high tx sites
• For physical removal:
– Hello…you’re on the street with a hot swap SCSI
disk and nowhere to put it
Settings – AI Block Size
• 16 Kb
– No brainer
– Do it before activating AI
• DB startup parameter
• Depends on your tx volume
• Start with 25-50 and monitor Buffer not avail
in promon – R&D – 2 – 6.
Helpers - AIW
• Another no-brainer
• Enterprise DB required
$ proaiw <db>
1. Disable AI
2. Delete the existing variable length extents
3. Add 4 fixed length 50 Mg AI extents
4. Change AI block size to 16 Kb
5. Change AIW=“yes” in go.sh
6. Add –aibufs 50 in server.pf
Compare results
ATM Workshop – Run Results
• No AI
Cl Time Trans Tps Conc Avg R Min R 50% R 90% R 95% R Max R
--- ---- ------ ------ ----- ----- ----- ----- ----- ----- -----
50 900 309493 343.9 48.0 0.1 0.0 0.1 0.3 0.5 3.1
• No AI = 343.9 tps
• AI + fixed extent + AIW = 344.7
• Difference is “noise”
– I.e. there’s no difference
– And this is a high tx benchmark!
Questions?