Automated Job Monitoring in a High Performance Computing Environment

Robert F. Cromp
Gilad Suberri
Raytheon Company

[email protected]
[email protected]

Abstract The Automated Job Monitor (AJM) project is

developing a customizable automated system for
We are developing software that monitors High identifying and rectifying routine system/queuing
Performance Computing assets while users’ batch issues normally performed by system administrators.
jobs execute, and actively performs site-established The AJM continually monitors a heterogeneous
corrective actions to handle routine system/queuing compute environment to uncover potentially
issues normally performed by Unix administrators. bothersome conditions, applying site-specific
The Automated Job Monitor is independent of both corrective actions as a result. As the make-up of this
platform and queueing system, and is customizable for environment changes (due to the retirement of old and
numerous domains. addition of new platforms), the system needs to
gracefully adapt [2]. We have chosen to pursue a
production rule system so as to separate the business
1. Introduction logic of the computing environment and its scheduling
policies from the analysis strategy for detecting and
A typical high performance computing (HPC) resolving potential system discrepancies. We agree
environment is quite complex, providing hundreds of that this approach provides for flexibility, adaptability,
active users access to hundreds or thousands of and ease in overall use, as exposited in [1].
processors. Computational efficiency is best achieved
by running jobs in batch mode—dedicating resources 2. System Design
to a specific job for an established time, thus
minimizing the time spent swapping processes and The Environment contains all the real time data
large volumes of data in memory, and eliminating that the application will monitor. This includes, for
problems with coordinating multiple processors from example, all queued jobs, running processes, and file
one job’s time share step to its next. system information. To analyze this data, a snapshot
To assist in scheduling these jobs, system is taken of the Environment and the results are stored
administrators provide specific queues with fixed in Working Memory. To assure the data is current,
properties. The queuing system matches each job to Working Memory is updated with a fresh snapshot of
the appropriate queues based on user-supplied the Environment at a configurable frequency.
constraints. The Inference Engine monitors the Working
The system administors need to ensure that each Memory for changes. When a piece of information is
HPC platform sustains a high work load, and that each updated with a new value, the Inference Engine
submitted job eventually has its resources allocated so collects all rules from the Rule Set that apply to the
that it executes. High performance computing centers updated data. These rules contain an antecedent and a
typically have a large number of jobs pending, and a consequent and thus can be represented as “if x is true,
job is usually not reviewed by a system administrator then do y.” The set of applicable rules is called the
until either a user questions why it has not started, or a Conflict Set and is analyzed by the Conflict Resolver,
system administrator notices that it has been pending which selects which rules to fire and assures that no
an unusually long time. Pinpointing the reason can be conflicting rules are evaluated. Information about the
time-consuming, due to the complexity of the compute rules fired is logged to a History Database to assist in
environment and the multitude of scheduling policies gaining a better understanding of the system.
in play.

Proceedings of the International Conference on Autonomic Computing (ICAC’04)

0-7695-2114-2/04 $20.00 © 2004 IEEE
 2.1 Readers Working Memory if its Read Set contains the attribute
being updated. The Read Set of a rule is defined as
A Reader is the entity that transfers information the set of attributes that are being evaluated, or read
from the Environment into Working Memory. Each from, in the antecedent. The Conflict Set is a set of
Reader knows what system calls need to be executed rules such that the Read Sets of all of these rules
to acquire the data it is responsible for transferring, contain the attribute updated.
how to parse the information, and how frequently to A Write Set is also associated with each rule. The
update Working Memory. The Reader parses the data Write Set of a rule is defined as the set of attributes the
into one or more Working Memory Objects. These rule’s consequent will write to and therefore possibly
objects are simply a list of attribute-name/value pairs change. The Conflict Resolver can use the Write Sets
that define the Environment’s data. of the rules in the Conflict Set to decide which ones to
A scheduler executes the Readers at their evaluate.
specified frequencies. Two schemes have been An issue concerning the Write Set of a rule is as
developed for this. The first scheme executes each follows. Assume Rule1 and Rule2 are in the Conflict
Reader at its configured frequency, allowing the Set, with the Read Set(Rule1) = {a}, Read Set (Rule2)
Reader to complete its system call, parse the data = {a, b}, and the Write Set(Rule1) = {b}. If Rule 1 is
returned, and update Working Memory before the next evaluated first and changes the value of b, this could
Reader is executed. Though in most cases this scheme change the outcome of whether or not Rule 2’s
is sufficient, a possible issue arises if two Readers are antecedent is evaluated to true, though it is part of the
scheduled at the same frequency. If the first Reader to Conflict Set corresponding to attribute a, not attribute
execute takes x amount of time to complete execution, b. Therefore, any changes rules make explicitly to
the next Reader will be x amount of time late in its attribute values are “local” changes, until all the
execution. This could cause unfavorable results. applicable rules have been evaluated and then the
The Producer-Consumer scheme addresses this “global” Working Memory can be updated.
issue. In this method, the Reader’s job is split into two
sections. The Producer is responsible for the system 3. Configuring the System
call and storing the data returned in a buffer. The
Consumer takes the data stored by the Producer, We have developed a graphical user interface
parses it, and inserts it into Working Memory. The using wxPython to assist the user in configuring their
Consumer is constantly running and checking for new specific monitoring scenario. The Class Builder is
data in the buffer; the Producer runs in respect to the used to define the structure of all the Working
frequency of the Reader, and only executes the system Memory Objects. The Rule Builder configures the
call before returning control to the scheduler. A system’s rules. The user defines the antecedent of the
Producer’s system call does not even need to complete rule by referring to objects defined in the Class
running before control is returned to the scheduler, Builder along with common logical and mathematical
giving the application more accuracy in terms of operators. Finally, the Reader Builder creates and
executing Readers at their respective frequencies. configures the system’s readers. The Builder analyzes
A Reader could also be called on an as-needed the output for different ways to parse, including
basis. For instance, another part of the system may common delimiters and fixed width, and displays the
realize that a certain piece of information critical in different possibilities. After a parsing style is selected,
making a decision is missing from Working Memory. the user sets the different pieces of the parsed output
The AJM will attempt to find and execute a Reader to with their corresponding attribute names. After this
get this specific data. process is completed (one time only per command), an
XML representation of the reader is created.
2.2 Rules
References
Rules are composed of an antecedent and a
consequent. An example would be, “If a running job [1] Chan, H. and B. Arnold, “A policy based system to
is using more processors than it requested, then kill the incorporate self-managing behaviors in applications,”
Companion of the 18th Annual ACM SIGPLAN
job and notify the system administrator of the job’s Conference on Object-oriented Programming,
details via email.” When an attribute’s value changes, Systems, Languages, and Applications, 94-95, 2003.
applicable rules are evaluated, thus allowing the AJM [2] Kephart, J. O. and D. M. Chess, “The vision of
to perform site-established corrective actions. A rule autonomic computing,” Computer, 36, 1, 41-50, 2003.
is considered applicable to an updated attribute of

Proceedings of the International Conference on Autonomic Computing (ICAC’04)

0-7695-2114-2/04 $20.00 © 2004 IEEE