Exploring Onstat - An Ibm Informix Ids Utility: Advanced Datatools Corporation
Exploring Onstat - An Ibm Informix Ids Utility: Advanced Datatools Corporation
Lester Knutsen
Advanced DataTools Corporation
Onstat – monitor IDS server
operations
• Onstat utility reads shared-memory structures
and provides statistics about the database
server at the time that the command executes.
• The contents of shared memory might change
as the onstat output displays.
• The onstat utility does not place any locks on
shared memory, so running the utility does not
affect performance.
• Onstat is a key utility to monitor the
performance of your IDS server
2
Informix IDS Shared Memory
•Control Tables
Resident Memory Segment •Buffers
LRU Page Page
•Dictionary cache
Virtual Memory Segment •Working Storage
•Sort Space
3
Onstat options:
onstat [ -abcdfghklmpstuxzBCDFRX ]
[ -i ]
[ -r [<seconds>] ]
[ -o [<outfile>] ] [ <infile> ]
-i Interactive mode
-r Repeat every <seconds> seconds (default: 5)
-o Put shared memory into specified dump file
<infile> Read shared memory information from specified
dump file
-z Zero profile counts
4
Key Onstat options:
• -d Print spaces and chunks
• -k Print locks
• -l Print logging
• -m Print message log
• -p Print profile
• -T Print TBLspaces
• -u Print user threads
• -D Print spaces and detailed chunk stats
• -F Print page flushers
• -R Print LRU queues
5
Key Onstat –g options
(New monitoring commands)
• ath Print all threads
• rea Print ready threads
• sch Print VP scheduler statistics
• seg Print memory segment statistics
• iof Print disk IO statistics by chunk/file
• ses [<session id>]Print session information
• sql [<session id>]Print SQL information
• mgm Print Memory Grant Manager information
6
Current status of Server: onstat -
lester@merlin >onstat -
lester@merlin >onstat -
shared memory not initialized for INFORMIXSERVER 'merlindb'
lester@merlin >
7
Onstat Header Information
Informix Dynamic Server 2000 Version 9.21.UC6 -- On-Line -- Up 2 days 23:48:57 --
3694592 Kbytes
8
Mode of Server
• Off-Line Mode (does not show in header)
• Quiescent Mode
• On-Line Mode
• Read-Only Mode (DR Only)
• Recovery Mode
• Shutdown Mode
9
Reason when Server is blocked
• CKPT - Checkpoint
• LONGTX - Long transaction
• ARCHIVE - Ongoing storage-space backup
• MEDIA_FAILURE - Media failure
• HANG_SYSTEM - Database server failure
• DBS_DROP - Dropping a dbspace
• DDR - Discrete data replication (IDS)
• LBU - Logs full high-watermark
10
Server Profile:
onstat -p
Informix Dynamic Server 2000 Version 9.21.UC6 -- On-Line -- Up 2 days 23:48:57 --
3694592 Kbytes
Profile
dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
412782302 493730321 2917750938 85.85 80916102 88592926 705858905 88.54
11
Informix IDS Shared Memory
•Control Tables
Resident Memory Segment •Buffers
LRU Page Page
•Dictionary cache
Virtual Memory Segment •Working Storage
•Sort Space
Page
12
Page Gets Read into Memory by a Select
•Control Tables
Resident Memory Segment •Buffers
LRU Page Page
•Dictionary cache
Virtual Memory Segment •Working Storage
•Sort Space
Page
13
When all Buffers are full, Least Recently Used (LRU)
Page is discarded to make room for more Data
•Control Tables
Resident Memory Segment •Buffers
LRU Page Page
•Dictionary cache
Virtual Memory Segment •Working Storage
•Sort Space
Page Page
Page Page Page
Page
14
When a user updates a page, it is marked as dirty and must be
written out to disk before it is discarded.
•Control Tables
Resident Memory Segment •Buffers
Updates
LRU Page Page
UpdatesLRU Page Page
•Dictionary cache
Virtual Memory Segment •Working Storage
•Sort Space
Page Page
Page Page Page
Page
15
LRU Queues manage writing data to disk in the background
when there is idle time based on LRU_MAX_DIRTY and
LRU_MIN_DIRTY ONCONFIG
Resident Memory values •Control Tables
Segment •Buffers
Updates
LRU Page Page
UpdatesLRU Page Page
•Dictionary cache
Virtual Memory Segment •Working Storage
•Sort Space
Page Page
Page Page Page
Page
16
When all buffers are Dirty, the server must STOP all processing and
perform a Foreground Write
•Control Tables
Resident Memory Segment •Buffers
Updates
LRU Page Page
All Activity is Blocked
•Dictionary cache
Virtual Memory Segment •Working Storage
•Sort Space
Page Page
Page Page Page
Page
17
When a buffer is written to disk, it is marked as clean and may
be discarded if needed.
•Control Tables
Resident Memory Segment •Buffers
Updates
LRU Page Page
UpdatesLRU Page Page
•Dictionary cache
Virtual Memory Segment •Working Storage
•Sort Space
Page Page
Page Page Page
Page
18
A Standard (7.X) Checkpoint writes all Dirty Buffers to Disk
•Control Tables
Resident Memory Segment •Buffers
Updates
LRU Page Page
UpdatesLRU Page Page
•Dictionary cache
Virtual Memory Segment •Working Storage
•Sort Space
Page Page
Page Page Page
Page
19
A Fuzzy (9.21) Checkpoint flags all Dirty Buffers and writes them to
Disk in the background without blocking activity
•Control Tables
Resident Memory Segment •Buffers
Updates
LRU Page Page
UpdatesLRU Page Page
•Dictionary cache
Virtual Memory Segment •Working Storage
•Sort Space
Page Page
Page Page Page
Page
20
After a Checkpoint, all Buffers are Clean and the cycle
Starts over again
•Control Tables
Resident Memory Segment •Buffers
LRU Page Page
•Dictionary cache
Virtual Memory Segment •Working Storage
•Sort Space
Page Page
Page Page Page
Page
21
Key Elements of onstat -p
22
Key Ratios Calculated from
Onstat -p
• Kbytes read and written per minute/hour
• Buffer turnover ratio per minute/hour
• Buffer wait ratio
• Read Ahead Utilization
23
Display message log file:
onstat -m
lester@merlin >onstat -m
INFORMIX-OnLine Version 9.2X.XXX -- On-Line -- Up 7 days 12:41:12 -- 10656
Kbytes
Message Log File: /u3/informix/online1.log
21:46:51 Checkpoint Completed: duration was 8 seconds.
21:52:00 Checkpoint Completed: duration was 9 seconds.
21:57:09 Checkpoint Completed: duration was 8 seconds.
22:00:42 Logical Log 20 Complete.
22:00:43 Process exited with return code 1: /bin/sh /bin/sh -c
/u3/informix/log_full.sh 2 23 "Logical Log 20 Complete." "Logical Log 20 Complete."
22:02:17 Checkpoint Completed: duration was 8 seconds.
24
Using “tail -f” to continuously
show the end of message log file
• Note: I like to have the OnLine log file always display in one of
my windows on screen. The trick to doing this is to use the
UNIX "tail" command with the "-f" option. This continually reads
the last lines of a file as it is appended to. On my system I run
the following command to continually monitor this log:
tail -f $INFORMIXDIR/online.log
25
User status: onstat -u
26
User status: onstat -u Flags
Flags in position 1
B - Waiting on a buffer
C - Waiting on a checkpoint
G - Waiting on a logical log buffer write
L - Waiting on a lock
S - Waiting on a mutex
T - Waiting on a transaction
Y - Waiting on a condition
X - Waiting on a transaction rollback
Flags in position 2
* - Transaction active during I/O error
Flags in position 3
A - Dbspace backup thread
B - Begin work
P - Prepared for commit work
X - TP/XA prepared for commit work
C - Committing work
R - Rolling back work
H - Heuristically rolling back work
27
User status: onstat -u Flags
Flags in position 4
P - Primary thread for a session
Flags in position 5
R - Reading call
X - Transaction is committing
Flags in position 6
None
Flags in position 7
B - Btree cleaner thread
C - Cleanup of terminated user
D - Daemon thread
F - Page flusher thread
M - ON-Monitor user thread
28
Logical Logs status: onstat -l
Physical Logging
Buffer bufused bufsize numpages numwrits pages/io
P-1 8 128 34175115 268502 127.28
phybegin physize phypos phyused %used
10003f 500000 108581 8 0.00
Logical Logging
Buffer bufused bufsize numrecs numpages numwrits recs/pages pages/io
L-3 0 64 1419836 62228 8468 22.8 7.3
Subsystem numrecs Log Space used
OLDRSAM 1419836 113663544
29
Logical Logs status:
onstat -l Flags
The flags column provides status information about each log.
A - Newly added, must run an archive before they can be used
B - Backed up to tape or "/dev/null"
C - Current logical log file
F - Free and available for use. You will rarely see this flag
as logs are not marked as free until right before they
are needed.
L - Last checkpoint is in this logical log
U - Used logical log, it may be free if it is backed up and
contains no active transactions.
One way to tell which logs can be reused is to use "onstat -l" with
"onstat -x" to display all active sessions.
30
Display transactions: onstat -x
This option displays all current transactions. The most useful column is "log
begin". This tells you in which logical log a transaction started. This
may be used with the "onstat -l" command to determine which logs are free
and may be reused. Find the earliest logical log number in the column "log
begin". This tells you which logical log has the earliest active
transaction. Any logical logs that are backed up before the log with the
earliest transaction will be automatically reused by OnLine.
lester@merlin >onstat -x
INFORMIX-OnLine Version 9.2X.XXX -- On-Line -- Up 01:21:38 -- 10656
Transactions
address flags userthread locks log begin isolation retrys coordinator
a2f4384 A---- a2d1118 0 0 NOTRANS 0
a2f44a8 A---- a2d1558 0 0 COMMIT 0
a2f45cc A-B-- a2d1118 2 21 NOTRANS 0
6 active, 128 total, 7 maximum concurrent
31
Display locks: onstat -k
WARNING: If you have a large number of LOCKS defined in your ONCONFIG file and many
users you could see thousands of rows from this command.
lester@merlin >onstat -k
INFORMIX-OnLine Version 9.2X.XXX -- On-Line -- Up 7 days 12:53:31 -- 10656
Locks
address wtlist owner lklist type tblsnum rowid key#/bsiz
a103e74 0 a2d2218 0 HDR+S 100002 20a 0
1 active, 20000 total, 16384 hash buckets
32
Who owns a lock
33
What table is locked?
database selected
tabname tblsnum
genjournal 0x0010009E
gjsum 0x0010009F
34
What table is locked?
35
Types of locks
36
Types of locks Flags
HDR - Header
B - Bytes lock
S - Shared lock
X - Exclusive
I - Intent
U - Update
IX - Intent-exclusive
IS - Intent-shared
SIX - Shared, Intent-exclusive
37
Dbspaces and chunks status:
onstat -d
lester@merlin >onstat -d
Dbspaces
address number flags fchunk nchunks flags owner name
a2ce100 1 1 1 1 N informix rootdbs
a2ce508 2 1 2 1 N informix dbspace1
a2ce578 3 1 3 1 N informix dbspace2
a2ce5e8 4 1 4 1 N informix dbspace3
4 active, 2047 maximum
Chunks
address chk/dbs offset size free bpages flags pathname
a2ce170 1 1 0 250000 62047 PO- /u3/dev/rootdbs1
a2ce280 2 2 0 10000 9587 PO- /u3/dev/dbspace1
a2ce358 3 3 0 10000 9947 PO- /u3/dev/dbspace2
a2ce430 4 4 0 10000 9947 PO- /u3/dev/dbspace3
4 active, 2047 maximum
38
onstat -d Flags
The "flags" for Dbspaces are: The "flags" for Chunks are:
Position 1 Position 1
M - Mirrored Dbspace P - Primary
N - Not Mirrored Dbspace M - Mirror
Position 2 Position 2
X - Newly mirrored O - On-line
P - Physical recovery D - Down
underway
X - Newly mirrored
L - Logical recovery underway
I - Inconsistent
R - Recovery underway
Position 3
Position 3
B - Blobspace Dbspace
B - Blobspace
T - Temporary Dbspace
39
Disk I/O: onstat -D
lester@merlin >onstat -D
Dbspaces
address number flags fchunk nchunks flags owner name
a2ce100 1 1 1 1 N informix rootdbs
a2ce508 2 1 2 1 N informix dbspace1
a2ce578 3 1 3 1 N informix dbspace2
a2ce5e8 4 1 4 1 N informix dbspace3
4 active, 2047 maximum
Chunks
address chk/dbs offset page Rd page Wr pathname
a2ce170 1 1 0 36563 179558 /u3/dev/rootdbs1
a2ce280 2 2 0 3 0 /u3/dev/dbspace1
a2ce358 3 3 0 2 0 /u3/dev/dbspace2
a2ce430 4 4 0 2 0 /u3/dev/dbspace3
4 active, 2047 maximum
40
Writing to Disk - onstat -F
lester@merlin >onstat -F
41
Writing to Disk - onstat -F
• Foreground writes occur when the Server needs a buffer and must
interrupt processing to flush buffers to disk to free a buffer. These
are the least desirable type of writes.
• Chunk writes occur at checkpoints, and all dirty buffer pages are
written to disk. The more dirty pages, the longer a checkpoint will
take. Checkpoint writes are sorted and optimized, but the longer a
checkpoint is, the longer it will block user activity. Checkpoint
writes are best for batch systems.
42
List all threads: onstat -g ath
43
List Virtual Processor status:
onstat -g sch
This option provides the means to identify which "oninit" UNIX process
corresponds to which Informix Server server Virtual Processor.
VP Scheduler Statistics:
44
List SQL statement types:
onstat -g sql
• This is the most interesting of the new options. This option allows you to
drill down and see the actual SQL statement that a user is executing.
45
List SQL statement for a user:
onstat -g sql sid
lester@merlin >onstat -g sql 264
46
List users sessions: onstat -g ses
47
Other Onstat Options
onstat -l -r 10
onstat -z
48
For more information
• International Informix User Group Software
Repository – http://www.iiug.org
• Mark Scranton’s web site
http://www.markscranton.com/informix
• IBM Informix Documentation
• IDS Administrators Reference Manual
49
Advanced Training Options
• Advanced Informix IDS Performance Tuning
and Optimization
– March 31 - April 4, 2003
– An advanced course for experienced Informix
DBAs.
• Informix IDS for Database Administrators
– February 10-14, 2003
– Configuring, managing, and tuning the Informix IDS
database server.
50
Lester Knutsen
Advanced DataTools Corporation
Web site: www.advancedatatools.com
Email: [email protected]
Lester Knutsen
Advanced DataTools Corporation
Email: [email protected]
52