2010-10-11 1

SKY-IMS Software Architecture

and Maintenance Training
Airports Oujda, Fes, Errachidia,
Al Hoceima

Stockholm
December, 2008
2010-10-11 2

AJAX = Asynchronous JavaScript and XML

Basic scenario
2010-10-11 3

SKY-IMS
• Integrated Meteorological System.
• Designed for 24 x 7 unattended operation.
• Fully compliant with ICAO & WMO regulations and
recommendations, open for adjustment with
national practices.
• Complex multipurpose software:
– Airport Weather Observation Systems
– Synoptic and Climatological monitoring
– Gamma radiation monitoring
– Marine meteorology
2010-10-11 4

Latest Technologies
Field proven sensors:
• NGDL750 Automatic Weather Station with sensors
• Intel based PC workstation running SKY-IMS software.
• Multiplatform: Microsoft Windows ® or Linux ®.
• Industry proven technologies: Java, XML, relational
databases (PostgreSQL).
• Data processing and archiving based on Hibernate and
SQL.
• Database based system configuration.
• Localization to national languages possible (2 languages
already available).
2010-10-11 5

Fully integrated web server

• Web user interface as primary for both local
and remote users.

• Meteo weather web server providing fully

integrated, remote displays of SKY-IMS
weather data over a vast array of network
environments, including the Internet.
2010-10-11 6

SKY-IMS Administration and Maintenance

Administrator skills assumed:

1. Experience with a PC hardware and
electronics.
2. Advanced experience with OS Linux
3. Basic experience with OS Linux
administration.
4. Basic experience with network administration
(TCP/IP networking).
5. Good knowledge of English
2010-10-11 7

SKY-IMS Administration Meta-Rules

Basic Administration Rules:
1. If it works, do not touch it.
2. If it does not work:
1. Read the documentation;
2. Think over the problem;
3. Consult SMMI AB if necessary;
4. Make sure you are able to restore current state of
system from installation media, backup media, etc.
before you start;
5. Act.
2010-10-11 8

Heartbeat
2010-10-11 9

DRBD - Distributed Replicated Block Device

2010-10-11 10

SKY-IMS Architecture - Processes

JAVA service
Wrapper
DRBD running running
as LINUX tomcat web
writer
daemon server
application
Starts Auxiliary
Starts programs run
OpenAIS/ from cron:
Pacemaker vacuum
running as SKY-IMS
LINUX daemon Application
Stores
Starts data
Retrieves
PostGreSQL data FireFox web browser
provides user with
database
access to web
running on interface
main server
2010-10-11 11

SKY-IMS directory structure

• SLES defualt directories
• /usr/lib64/jvm/java – the directory with Java
Virtual Machine
• /var/log/tomcat6 – log files organized in time tree.
• /srv/tomcat6 – Apache TOMCAT application server
• /var/lib/pgsql – directory with database
• /opt/writer – The writer application
• /opt/writer/logs – log files oranized in time tree
• /opt/writer/raws – Raw data files
2010-10-11 12

SKY-IMS main menu

Server Time (UTC)

Menu

Client UTC
Client local time
time
2010-10-11 13

Cluster Management Screen

Displays MAIN and
STANDBY status of
computers and:
•logged user
•system uptime
•memory status
•disc status
2010-10-11 14

SKY-IMS System Startup

1. Linux boots
2. Daemons drbd and openAIS/pacemaker is started automatically
(postgresql and writer is started from pacemaker)
3. Daemon tomcat6 is tsarted automatically
1. Database connection WEB-service is started
2. Buisness logic WEB-service is started
3. SKY-IMS web application is started

Manual start/stop of tomcat6 daemon:

• through /etc/init.d/tomcat6 start, /etc/init.d/tomcat6 stop scripts

Similar for manual start/stop of openais

2010-10-11 15

SKY-IMS System Restart

1. Restart:
– Restart through web interface reboots the entire
machine
2. Manual restart by command (restarts
application)
– “/etc/init.d/tomcat6 restart”
3. Manual stopping by command (stops
application)
– “/etc/init.d/tomcat6 stop”
2010-10-11 16

SKY-IMS Processes
1. HA cluster process – openAIS
2. Database writer – writer
3. Web interface with application – Tomcat6
2010-10-11 17

Troubleshooting – SKY-IMS is not starting

2010-10-11 18

Fully integrated web server

• Web user interface as primary for both local
and remote users.
• Meteo weather web server providing fully
integrated, remote displays of SKY-IMS
weather data over a vast array of network
environments, including the Internet.
2010-10-11 19

Fully integrated web server

TCP Port 8080

Standard connection

TCP Port 8443

Standard connection
2010-10-11 20

AJAX = Asynchronous JavaScript and XML

Basic scenario
Client TOMCAT running
SKY-IMS
HTTP request: index.jsp
Tomcat
HTTP response: default servlet
index.jsp compiled

XML HTTP request SKY-IMS response

for data Time
servlet
XML HTTP response:
XML with data

XML HTTP request

for data SKY-IMS response
servlet
XML HTTP response:
XML with data
2010-10-11 21

FireFox Browser
FireFox browser is not IMS Software !

IMS Software runs at background even if:

•No user is logged to Linux;
•A user is logged to Linux, but no browser is
running.
2010-10-11 22

Web Interface Troubleshooting

• Browser is still showing old version interface after update:
– clear the browser local cache by choosing Tools -> Clear Recent
History -> Clear Now
• The user cannot log, although the username and
password is correct:
– make sure the cookies are enabled in the browser
• Opening of more than two windows in FireFox may result
in delayed update of screens due to client (FireFox) limits.
• The user session expires automatically after 30 minutes of
inactivity (configurable), explicit logout is recommended,
however
2010-10-11 23

SKY-IMS Database
• PostgreSQL 8.4 database server running on main server
• Database physically located in /var/lib/pgsql directory
(which is linked to /dev/drbd0 directory)
• Database running as postgresql service managed by
pacemaker.
• SQL Server available at TCP port 5432.
• Database available as “meteoska” at main server.
• Access to database through SKY-IMS Web Interface:
– Graphical Data
– Archives
2010-10-11 24

SKY-IMS Database
SKY-IMS Database stores:
• measured, manually entered and computed
data
• WMO codes created locally (SYNOP, METAR,
CLIMAT)
• Messages exchanged (sent, received) through
exchange networks (GTS, AFTN)
• User accounts
2010-10-11 25

SKY-IMS Database
IMS Database does not store:
• record of raw communication through
communication channels;
• Current data received from other stations for
preview;
• log files.
2010-10-11 26

SKY-IMS Database Dump

The SKY-IMS Database can be dumped for
backup purposes by command (typed into
single line):

# /usr/bin/pg_dump -h localhost
–p 5432 -U ims -F c -f
ims.backup ims –v
2010-10-11 27

Database: Vacuum Operation

VACUUM Operation:
• Reclaiming of database space for reuse after
delete
• Run as task scheduled from cron daemon.
2010-10-11 28

Security, User Access

• Login = login inside SKY.IMS managed by
Spring security
• Lazy role-based authentication scheme:
– TOMCAT is a container of resources (web pages,
wav files, etc.)
– access to some resources requires authentication
– user is prompted to authenticate the first time
he/she requires a protected resource
2010-10-11 29

Login
Client
Server

HTTP request to protected resource

Spring security login
procedure
HTTPS login form request
to authentication

HTTPS: username, Time

password

HTTPS response: Spring security login

protected resource procedure
2010-10-11 30

Roles
• Recommendation: each user has its own
personal account
• Each user has assigned some roles:
– operator
– atc (not used)
– admin
• User not logged in:
– read access to current and historical data
– no right to change anything
2010-10-11 31

Roles
User with role operator:
• Rights as not logged user
• Right to create WMO Codes:
– SYNOP
– METAR
• Right to change his password
2010-10-11 32

Roles
User with role admin:
1. Rights as not logged user
2. Right to change IMS settings:
– Switch on/off channels
– Change system parameters
3. Right to add users, assign roles.

Admin is not Linux root !

2010-10-11 33

Passwords
1. Usernames and passwords are stored in SKY-
IMS database.
2. Empty database = nobody can log in.
3. Passwords are encrypted. Admin\user can
change password, but cannot decrypt
password.
2010-10-11 34

Troubleshooting: What if no accounts are in

database ?
1. Linux root can initialize default admin
account with default password by
/opt/writer/setDefaultAdmin.sh
2. Admin can log in using default password
and initialize accounts. It is strongly
recommended to change password for the
default account during first login.
2010-10-11 35

Software Maintenance
• Maintenance of Linux operating system
– Automatic updates
– Antivirus protection
• Maintenance of IMS software:
– Preventive checks
– Troubleshooting
– Updating
2010-10-11 36

Software – Failure prevention\recovery

• Memory – if software allocates more memory
than allowed (limit 384 Mb), automatic
restart.
• Automatic deleting of the old files on the disk.
• Automatic deleting of the old records in the
database.
• Optional visual \ audio alerts at the SKY-IMS
workstations
2010-10-11 37

Software Maintenance
Checking of Alerts:
• If some unusual ERROR/WARNING is not reported by the
system

Checking of Communication Screen:

• If all communication channels indicate normal (green status)

Checking of Cluster Management Screen:

• If the uptime is not low (indicates restart of system);
• If the memory is not consumed;
• If the disc is not consumed.
2010-10-11 38

Troubleshooting – Disc Consumed

If the disc is consumed:
• make sure the non-SKY-IMS and non-system files do not
consume the disc space, if they consume, erase them.
• make sure the log files and database deletion limits are
not too high (storing of data for years, etc.). If the
storage limits are to high:
– Make the limits lower
– Erase the log files manually from
/var/log/tomcat6/drirectroy
– Do not try to erase database data manually, start SKY-IMS
and wait until the data are automatically erased
2010-10-11 39

High Availability SKY-IMS – Dual Cluster

Serial line

10.0.0.1 Dedicated LAN 10.0.0.2

Two servers running in hot failover mode.

Automatic failover between main and standby server
not affecting the system operation.
Heartbeat over dedicated LAN interface and over
serial line(redundant path option).
Data mirroring over dedicated LAN:
– database synchronization by DRBD
– TOMCAT cluster - session replication
2010-10-11 40

Heartbeat – Linux High

Availability Project linux-ha.org
Heartbeat is started when the server is started.
Heartbeat is shut down when the server is shut down.
When heartbeat is running, the node is considered to be alive.
Configuration files:
/etc/ha.d/ha.cf
– logfile, debugfile
– heartbeat serial line path
– serial line, baudrate
– heartbeat Ethernet unicast
– interface, UDP port
/etc/ha.d/haresources
IP failover by ARP spoofing technique (broadcasts of ARP packets saying
“shared IP address maps to my MAC address”).
Command line utilities: /usr/bin/cl_status
2010-10-11 41

MAIN Server
1. common IP address is mapped to MAIN server as eth0:0:
skyims1 : # ifconfig
eth0 Link encap:Ethernet HWaddr 00:D0:C9:9D:1C:D7
inet addr:192.168.1.100 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::2d0:c9ff:fe9d:1cd7/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:9363389 errors:0 dropped:0 overruns:0 frame:0
TX packets:10117004 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3089747611 (2946.6 Mb) TX bytes:3472538748 (3311.6 Mb)
Interrupt:177
eth0:0 Link encap:Ethernet HWaddr 00:D0:C9:9D:1C:D7
inet addr:192.168.1.102 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:177

2. PostgreSQL database server is started on the MAIN server

2010-10-11 42

DRBD
Distributed Replicable Block device – mirroring disk
partitions over LAN connection
Configuration: /etc/drbd.conf
Check current state: cat /proc/drbd
DRBD service: /etc/init.d/drbd
NOTE:
If there is problem with LAN or DRBD cannot get
information about state of devices during boot
process, administrator have to put ‘yes’ to start up
console
2010-10-11 43

Pacemaker log
Important keywords:
• heartbeat – process, when a node in cluster announces its
correct operation.
• heartbeat link – serial link or ethernet card, over which
heartbeat is performed. For increased reliability, 2 links on
different media are recommended.
• resource – a resource (service or IP address) to be shared
among nodes of cluster.
• split brain – situation, when more than one node owns
resources (usually when heartbeat links are broken, or
keepalive timeout is too small).
• node status – active or dead.
2010-10-11 44

Remote Maintenance by VNC

• X11VNC server installed during IMS
installation.
• X11VNC starts automatically during boot:
• TCP server listening on port 5900
• X11VNC can be started manually by server by
command:
• /usr/bin/x11vnc –display :0 –forever –bg –
passwd PASSWORD
2010-10-11 45

Network – TCP/IP Summary

Ports which should be available to public:
• TCP 8080: TOMCAT HTTP port
• TCP 8443: TOMCAT HTTPS port
• TCP 22: ssh for remote maintenance
• TCP 5900: VNC for remote maintenance
Ports which should be available locally within cluster:
• TCP port 5432: PostgreSQL
• TCP port 4001: TOMCAT cluster
• TCP port 7789: DRBD synchronization
• UDP port 694: heartbeat
2010-10-11 46

Updating
• Updates are provided usually SMMI AB
either as .rpm files or .tgz files with
instructions how to install the update.
• SMMI AB always provides the step-by-step
installation instructions (although they are
usually very short: “run the command “tar –C
/ -xvzf SKYIMS_XXX.tgz”).
• For running of the update files the user must
be logged in as user root