Useful UNIX commands for WebSphere & Portal

Administrator
These are some useful commands that i came across when administrating WebSphere
Application Server or WebSphere Portal Server. Note these commands are specifically written
for LINUX but most of them would for other UNIX platforms like Solaris, AIX and other flavours
of LINUX like SUSE or REDHAT.

Performance related Commands to monitor system performance

-------------------------------------------------------------------------
1) Find the process that uses most CPU
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -2
(eg) output of the above command when websphere server process is utilizing the most of the
CPU

%CPU PID USER COMMAND

71.7 31237 root /cust/IBM/WebSphere/AppServer/java/bin/java
-Xbootclasspath
/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere

/AppServer/java/jre/lib/ext/ibmext.jar
-Dwas.status.socket=54859 -classpath

/cust/IBM/WebSphere/AppServer/profiles/AppSrv02/properties:/cust/IBM/WebSphere/AppSer

2) Find the last 10 process that use the most CPU

ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10

%CPU PID USER COMMAND

2.1 6375 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath
/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServ
status.socket=2132 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties
0.3 13886 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath
/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServ
status.socket=25042 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/propertie
0.2 17271 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath
/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServ
status.socket=9606 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties
0.2 17871 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath
/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServ
status.socket=9606 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties
0.2 17713 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath
/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServ
status.socket=9140 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties
0.1 18499 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath
/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServ
status.socket=11309 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/propertie
0.1 17504 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath
/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServ
status.socket=8969 -classpath /cust/IBM/WebSphere/AppServer/profiles/Dmgr01/properties:/

3)Find cpu usage

mpstat -P ALL or mpstat

Linux 2.4.21-27.ELsmp (machine1.www.com) 02/02/2008

11:55:00 AM CPU %user %nice %system %iowait %irq %soft %idle

intr/s
11:55:00 AM all 1.51 0.00 0.41 0.04 0.01 0.09 97.94
184.92
11:55:00 AM 0 1.50 0.00 0.41 0.04 0.01 0.11 97.93
123.76
11:55:00 AM 1 1.51 0.00 0.42 0.04 0.00 0.07 97.95
61.16

4)Find CPU usage with the process listing in runtime and much more statistics in terms of
memory, virtual memory, etc.
top

12:00:20 up 118 days, 15:16, 2 users, load average: 0.07, 0.02, 0.00
572 processes: 570 sleeping, 2 running, 0 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 2.7% 0.0% 1.1% 0.0% 0.0% 0.0% 95.9%
cpu00 1.9% 0.0% 0.3% 0.0% 0.0% 0.0% 97.6%
cpu01 3.5% 0.0% 1.9% 0.1% 0.0% 0.0% 94.2%
Mem: 11819592k av, 5088092k used, 6731500k free, 0k shrd,
295304k buff
3998132k active, 682312k inactive
Swap: 2048276k av, 0k used, 2048276k free 1258592k
cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU
COMMAND
17719 root 16 0 306M 306M 3992 S 0.7 2.6 22:14 1 java
19358 root 16 0 2532 2532 896 R 0.5 0.0 0:00 1 top
17771 root 15 0 306M 306M 3992 S 0.4 2.6 3:48 0 java
18058 root 16 0 458M 458M 4032 S 0.1 3.9 12:55 0 java
1 root 15 0 512 512 452 S 0.0 0.0 1:19 0 init
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0
migration/0

5)Continuous CPU usage monitoring and find the average CPU used.
a) sar -u 2 5
b) sar -o output.file 12 8 >/dev/null 2>&1 &
c) nohup sar -o output.file 12 8 >/dev/null 2>&1 &

Linux 2.4.21-27.ELsmp (machine1.web.com) 02/02/2008

12:04:34 PM CPU %user %nice %system %iowait %idle

12:04:36 PM all 0.25 0.00 0.00 0.00 99.75
12:04:38 PM all 0.00 0.00 0.50 0.25 99.25
12:04:40 PM all 7.00 0.00 7.25 0.00 85.75
12:04:42 PM all 21.05 0.00 1.50 0.00 77.44
12:04:44 PM all 3.00 0.00 0.50 0.00 96.50
Average: all 6.26 0.00 1.95 0.05 91.74

Disk Usage related Commands to monitor Disk space

--------------------------------------------------------
1)Disk Usage on all mounts
df -h -T

Filesystem Type Size Used Avail Use% Mounted on

/dev/sda1 ext3 2.9G 785M 2.0G 29% /
/dev/sda7 ext3 57G 9.3G 45G 18% /cust
/dev/sda6 ext3 2.0G 82M 1.8G 5% /lc
none tmpfs 5.7G 0 5.7G 0% /dev/shm
/dev/sda5 ext3 2.0G 33M 1.8G 2% /tmp
/dev/sda3 ext3 2.0G 64M 1.8G 4% /var

2)prints just one line with the total size of the directory, the below example show the size of
the /usr/IBM/Websphere WebSphere Portal Server directory.
du -ch | grep total
bash-2.05b# du -ch | grep total
6.3G total

Network related Commands

-------------------------------------
1)
a) dig (domain information groper) - is a flexible tool for interrogating DNS name servers.
b) nslookup - Queries a name server for a host or domain lookup.
c) host - DNS lookup uility.

The below example shows the information like CNAME , Aliases, ipaddres , etc about
www.google.com

a) dig <ip address or hostname>

bash-2.05b# dig www.google.com

; <<>> DiG 9.2.4 <<>> www.google.com

;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1088
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 7, ADDITIONAL: 0

;; QUESTION SECTION:
;www.google.com. IN A

;; ANSWER SECTION:
www.google.com. 585119 IN CNAME www.l.google.com.
www.l.google.com. 69 IN A 216.239.51.99
www.l.google.com. 69 IN A 216.239.51.104

;; AUTHORITY SECTION:
l.google.com. 60734 IN NS g.l.google.com.
l.google.com. 60734 IN NS b.l.google.com.
l.google.com. 60734 IN NS f.l.google.com.
l.google.com. 60734 IN NS c.l.google.com.
l.google.com. 60734 IN NS a.l.google.com.
l.google.com. 60734 IN NS e.l.google.com.
l.google.com. 60734 IN NS d.l.google.com.

;; Query time: 2 msec

;; SERVER: 10.5.50.184#53(10.5.50.184)
;; WHEN: Sun Feb 3 09:51:46 2008
;; MSG SIZE rcvd: 196
b) host <ip address or hostname>

bash-2.05b# host www.google.com

www.google.com is an alias for www.l.google.com.
www.l.google.com has address 216.239.51.104
www.l.google.com has address 216.239.51.99

c) nslookup <ip address or hostname>

bash-2.05b# nslookup www.google.com

Server: 10.5.50.184
Address: 10.5.50.184#53

Non-authoritative answer:
www.google.com canonical name = www.l.google.com.
Name: www.l.google.com
Address: 216.239.51.99
Name: www.l.google.com
Address: 216.239.51.104

2) List all network interfaces, the below example list two interfaces eth0 and eth1 and a
loopback.

infconfig -a

bash-2.05b# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:11:43:32:49:99
inet addr:10.2.150.74 Bcast:10.5.50.127 Mask:255.255.255.192
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:357854361 errors:0 dropped:0 overruns:0 frame:0
TX packets:383621806 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
 RX bytes:3168219417 (3021.4 Mb) TX bytes:3724305382 (3551.7
Mb)
Interrupt:17

eth1 Link encap:Ethernet HWaddr 00:11:43:32:49:9A

inet addr:10.5.154.138 Bcast:10.5.54.127
Mask:255.255.255.128
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:18

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:92809439 errors:0 dropped:0 overruns:0 frame:0
TX packets:92809439 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1454141744 (1386.7 Mb) TX bytes:1454141744 (1386.7
Mb)

3) To find more information about a network interface, the below example shows detail
information about interface eth0.

bash-2.05b# ethtool eth0

Settings for eth0:

Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full

1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: Not reported
Advertised auto-negotiation: No
Speed: 100Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Link detected: yes
File related commands
----------------------
1) find text in the matching files and show the filename with the pattern. The below example
shows the list of filenames and the location of the hostname in those files, which will be
helpful when changing hostnames for your websphere installation.

find . -name '*.xml' -print | xargs grep 'hostName' /dev/null

bash-2.05b# find /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/ -name 'server*.x

| more
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/por
ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http
.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="portalserver.example.com" endPointR
ndPoint_1130375480668 NamedEndPoint_1130375480669 NamedEndPoint_1130375480670 NamedEndPo
2 NamedEndPoint_1130375480673">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n20
serverindex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serve
er/schemas/5.0/serverindex.xmi" xmi:id="ServerIndex_1130386859873" hostName="n2034indchn
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n20
serverindex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serve
er/schemas/5.0/serverindex.xmi" xmi:id="ServerIndex_1130361145826" hostName="n2033indchn
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/por
verindex:ServerIndex xmi:id="ServerIndex_1" endPointRefs="NamedEndPoint_1 NamedEndPoint_
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n20
ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http
.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="n2011indchn2.example.com" endPointR
ndPoint_1130375717655 NamedEndPoint_1130375717656 NamedEndPoint_1130375717657 NamedEndPo
9 NamedEndPoint_1130375717660">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n20
x:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="ht
/5.0/serverindex.xmi" xmi:id="ServerIndex_1173388344070"hostName="n2033indchn2.example.c
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n20
x:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="ht
/5.0/serverindex.xmi" xmi:id="ServerIndex_1173388421900" hostName="n2034indchn2.example.
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/templates/servertypes/APPLICATION_
ndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.
erindex.xmi" xmi:id="ServerIndex_1" hostName="$(node.host.name)">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/backup/base/cells/portalserverNode
ml:<serverindex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:s
server/schemas/5.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="portalserver.exampl
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/backup/base/templates/servertypes/
ndex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex=
mas/5.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="$(node.host.name)">

2) Soemtimes you might run out of diskspace and want to clean up some files, so this command
will help you find all large files. In this example it will list all the files greater than 10 mb in
size.

find / -xdev -size +1024 -exec ls -al {} \; | sort -r -k 5

bash-2.05b# find /usr/IBM/WebSphere -xdev -size +1024 -exec ls -al {} \; | sort -r -k 5-

1 root root 1047553 Oct 26 2005
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/tranlog/n2110indch2Cell01/n2110indch2Node
-rw-r--r-- 1 root root 1047553 Oct 26 2005
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/tranlog/n2110indch2Cell01/n2110indch2Node
-rw-r--r-- 1 root root 1047553 Oct 26 2005
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/tranlog/n2110indch2Cell01/n2110indch2Node

3) When sending SystemOut.log or SystemErr.log or trace.log to your developers or to IBM

support Please use Tar and gzip on the fly to reduce the size of the file and the transfer time to
FTP,email or http upload.

To pack on the fly...

tar -cvf - FILE-LIST | gzip -c > FILE.tar.gz

To unpack on the fly..

gunzip < FILE.tar.gz | tar -xvf -

To pack on the fly and email the file to IBM Support

tar -cvf - SystemOut.log | gzip -c | uuencode SystemOut.tar.gz | mail -s "PMR
09921,227,000" -b "SystemOut.log for server crash " [email protected]

To find selective files and tar it up on the fly...

find /opt/IBM/WebSphere/PortalServer/log -name 'System*.log'| xargs tar -rvf
System_logs.tar

4) Sometimes you might encounter a situation for example when running xmlaccess or wsadmin
where you might want to write stdout to the file as well as to the console so that you have a
copy even when the standard output disappears because of the screen buffer size.

ls -l | tee > filename

xmlaccess.sh -in ExportRelease.xml -username wpsadmin -password wpsadmin -url
http://localhost:9080/wps/config | tee filename
./wsadmin.sh | tee wsadmin.log

5) find files that is recently updated which might be useful to see what changes are made to
the system recently (e.g) below command shows the files that are modified within the last 20
minutes(-mmin) and 20 days (-mtime)

find /usr/IBM -mmin -20

find /usr/IBM -mtime -20
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/dmgr/SystemOut.log
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/ffdc/dmgr_exception.log
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/activity.log
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/wstemp/events/eventbuffer0.ser
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/wstemp/events/eventbuffer1.ser
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/wstemp/events/eventbuffer2.ser
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/logs

6) Find a memory usage of the system.

free
cat /proc/meminfo

bash-2.05b# free
total used free shared buffers
cached
Mem: 11819592 4938416 6881176 0 298148
1440200
-/+ buffers/cache: 3200068 8619524
Swap: 2048276 0 2048276

bash-2.05b# cat /proc/meminfo

total: used: free: shared: buffers: cached:
Mem: 12103262208 5056954368 7046307840 0 305303552 1474797568
Swap: 2097434624 0 2097434624
MemTotal: 11819592 kB
MemFree: 6881160 kB
MemShared: 0 kB
Buffers: 298148 kB
Cached: 1440232 kB
SwapCached: 0 kB
Active: 3718616 kB
ActiveAnon: 2968776 kB
ActiveCache: 749840 kB
Inact_dirty: 789328 kB
Inact_laundry: 237596 kB
Inact_clean: 0 kB
Inact_target: 949108 kB
HighTotal: 11140992 kB
HighFree: 6681628 kB
LowTotal: 678600 kB
LowFree: 199532 kB
SwapTotal: 2048276 kB
SwapFree: 2048276 kB
Committed_AS: 6039100 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB

4) To repeat a command contimuously.

watch -d=10 -n 1 "netstat -a"

Every 1s: netstat -a| grep ESTABLISHED

Sun Feb 3 21:48:04 2008

tcp 0 0 n1212indchn2.www.com:9900 n1312indchn2.www.com:11530

ESTABLISHED
tcp 0 0 n1212indchn2.www.com:9353 n1312indchn2.www.com:11549
ESTABLISHED
tcp 0 0 n1212indchn2.www.com:9354 n1312indchn2.www.com:11550
ESTABLISHED
tcp 0 0 n1212indchn2.www.com:9201 n1312indchn2.www.com:11544
ESTABLISHED
tcp 0 0 n1212indchn2.www.com:9352 n1312indchn2.www.com:11545
ESTABLISHED