The Alarm Management Handbook - Second Edition: A Comprehensive Guide

PAS, 16055 Space Center Blvd, Suite 600, Houston, TX 77062

© 2006, 2010 by PAS

All rights reserved. First edition 2006

Second edition 2010

Printed in the United States of America by 360 Digital Books, Kalamazoo, MI 49009.

18 17 16 15 14 13 12 11 10   1 2 3 4 5

ISBN: 978-0-9778969-3-6

This book, or parts thereof, may not be reproduced in any form without permission.

The scanning, uploading, and distribution of this book via the Internet or via any other means without the permission of the publisher is illegal and punishable by law.

For the second edition of this book, my dedication remains the same!

For Pam,

my beautiful, patient, and adorable trophy wife of 34 years.

-Bill

To the console operator.

- Eddie

In Memoriam

Chris Uehlinger

1956 - 2005

In 2005, PAS lost President and Chief Operating Officer, Chris Uehlinger. With his death, PAS has lost a great friend and business partner, and the process controls industry worldwide has lost an admired industry leader and visionary.

Chris’ commitment to excellence helped shape the vision of improved process safety and reliability through advanced automation technologies.

Chris joined PAS in the spring of 2000 after a long and successful career as a vice president with Honeywell Process Solutions. At PAS, he continued to demonstrate the qualities that made him an exceptional leader: thoughtful and strategic thinking, personable leadership, and an extraordinary ability to grasp and communicate complex technological concepts. Chris' career began at Celanese after he earned his engineering degree from Texas A&I University (now known as Texas A&M University) in Kingsville, Texas. Chris will be greatly missed and fondly remembered.

- Eddie Habibi, PAS Founder and CEO

Acknowledgements

Many people have significantly contributed to the Alarm Management body of knowledge and deserve recognition and thanks for those contributions. Here are a few of these people.

First and foremost, the authors appreciate and acknowledge the efforts and dedication of PAS employees. Their tireless work has resulted in significant advancement of the state of the art of alarm management. Software and work processes have benefited from their leadership in hundreds of successful worldwide alarm management projects.

We also greatly appreciate the significant effort, quality of work, and patience Jennifer Hicks demonstrated in supporting and driving the team to deliver this book on a very tight schedule.

Since the first edition of this book, the ISA 18.2 Alarm Management standard was completed. This major effort was led by two people that deserve special recognition – the committee chairs Nick Sands of DuPont and Donald Dunn of Aramco.

Other alarm management contributors and supporters include Joseph Alford, Peter Andow, Alex Boquiren, John Campbell, Donald Campbell-Brown, Jim Cawood, Mike Clark, Kenneth Emigholz, Ralph Harris, Heinz Janiec, Les Jensen, Lokesh Kalra, Douglas Metzger, Yahya Nazer, Ian Nimmo, Patrick O'Donnell, Douglas Rothenberg, Bill Sheldon, Todd Stauffer, David Strobhar, and Lee Swindler.

Organizations are also advancing the cause of alarm management. The American Petroleum Institute is currently working on an Alarm Management Recommended Practice, led by Gary Medley of BP Pipelines. The Electric Power Research Institute (EPRI) has championed the cause of Alarm Management, with leadership provided by Wayne Crawford. And both EEMUA (the Engineering Equipment and Materials User Association), and the Abnormal Situation Management Consortium (ASM®) continue to work on advancing the cause of process safety and proper alarm management.

Chapter Listing

Introduction

Chapter 1     Alarm Management Best Practices: Highly Condensed

Chapter 2     The History and Nature of the Alarm Problem

Chapter 3     How Do You Justify Alarm Management?

Chapter 4     Common DCS and SCADA Alarm Display Capabilities – and Their Misuse

Chapter 5     Step 1: The Alarm Philosophy

Chapter 6    Step 2: Baseline and Benchmarking of Alarm System Performance

Chapter 7     Step 3: Alarm Bad Actor Analysis and Solution

Chapter 8     Step 4: Alarm Documentation and Rationalization

Chapter 9     Step 5: Alarm Auditing and Enforcement

Chapter 10   Step 6: Advanced, Real-Time Alarm Solutions

Chapter 11   Step 7: The Control and Maintenance of Alarm System Improvements

Chapter 12   Understanding and Applying ANSI/ISA-18.2: Management of Alarms for the Process Industries

Chapter 13   The Future of Alarm Management

Appendix 1   The Death of the Lightbox

Appendix 2   Alarm Data Types

Appendix 3   An Example Alarm Philosophy

Appendix 4   Operator Effectiveness

References

Index of Terms

Detailed Table of Contents

In Memoriam

Acknowledgements

Foreword

Introduction

Alarm Management Best Practices: Highly Condensed

1.1     The Alarm Problem

1.2     People Who Can Help

1.3     The ANSI/ISA-18.2-2009 Alarm Management Standard

1.4     Seven Steps to a Highly Effective Alarm System

1.5     Summary

The History and Nature of the Alarm Problem

2.1     How Did We Get In This Mess?

2.2     Where Are We Now?

2.3     Regulatory Agencies and Alarm Management

How Do You Justify Alarm Management?

3.1     Economic Justification of Alarm Management

3.2     Analogy to Advanced Process Control

3.3     Analogy to Safety Systems

3.4     Incident Analysis

3.5     Liability

3.6     Aging Workforce

3.7     Summary

Common DCS and SCADA Alarm Display Capabilities – and Their Misuse

4.1     DCS and SCADA Alarm System Capabilities

4.2     The Alarm Display

4.3     Custom Graphics – Alarm-Related Guidelines

4.4     The Nature of Alarm Priority

4.5     Alarm Priority and Color

4.6     Alarm Priority and Sound

4.7     Alarm Priority Distribution

4.8     Alarm Priority Distribution and Process Control Capability

Step 1: The Alarm Philosophy

5.1     The Alarm Philosophy Document

5.2     What Should Be An Alarm?

5.3     The Basic and Proper Principles of Alarming

5.4     Poor Alarm Practices

5.5     Alarm Philosophy Customization

5.6     Specific Alarm Design Considerations

5.7     Alarms Used to Prevent Harm to Personnel

5.8     Alarms from Instrument Malfunctions (Diagnostic Alarms)

5.9     Use and Abuse of Combination Alarms

5.10   Alarm Placement

5.11   Redundant Sensors, Voting, and Shutdown Systems

5.12   Safety System (ESD) Alarm Priority

5.13   Safety System (ESD) Bypass/Testing Alarms

5.14   External Devices Alarms – An Engineering Love Story

5.15   Alarms from Fieldbus - type Devices

5.16   Alarm Generation by Programs

5.17   Alarms to Initiate Manual Tasks

5.18   DCS System Diagnostic Alarms

5.19   Point and Program References to Alarms

5.20   Operator Messaging Systems

Step 2: Baseline and Benchmarking of Alarm System Performance

6.1     Operator Alarm Handling Capacity

6.2     Operator Span of Control and Multiple Operators

6.3     Alarms Are Not Created Equally

6.4     The History of Alarm Analysis

6.5     Alarm System Key Performance Indicators (KPIs)

6.6     Alarms per Day

6.7     Alarms per Ten Minutes

6.8     Alarm Floods

6.9     Alarms Likely to Have Been Missed

6.10   Most Frequent Alarms

6.11   Alarm Priority Distribution

6.12   Chattering Alarms

6.13   Alarms by Type

6.14   Alarm Suppression

6.15   Stale Alarms

6.16   Changes Requiring Management-of-Change Procedures

6.17   Analyses Involving Alarm Acknowledgement

6.18   The Alarm Management Champion

6.19   Alarm Performance Reports

6.20   Alarm Performance Numbers: Are You a Special Case?

6.21   Alarm Configuration Analysis

6.22   Configured Alarm Priorities by Alarm Type

6.23   Configured Alarm Ratio

6.24   Alarm System Performance Levels

6.25   Specific Alarm System Improvement Plans

6.26   Analysis of Operator Actions

6.27   Controller Mode Changes

6.28   Controller Setpoint Changes

6.29   Overall Operator Change Rate

6.30   Correlation of Operator Actions and Alarms

Step 3: Alarm Bad Actor Analysis and Solution

7.1     Common Alarm Problems and How to Solve Them

7.2     Expected Results from Bad Actor Resolution

7.3     Chattering and Fleeting Alarms

7.4     Alarm Deadband

7.5     Process Value Filtering and Alarms

7.6     Delay Time Analysis and Alarms

7.7     Other Frequent Alarms

7.8     Suppressed Alarms

7.9     Stale (Long-Standing) Alarms

7.10   Duplicate Alarms

7.11   Nuisance Bad Measurement Alarms

7.12   Summary

Step 4: Alarm Documentation and Rationalization (D&R)

8.1     D&R Overview

8.2     D&R Participation and Reference Information

8.3     Every Alarm Has a Constituency

8.4     The Alarm Rationalization Grids

8.5     Areas of Impact and Severity of Consequences

8.6     Special Guidelines: Probability

8.7     Special Guidelines: Multiple Failures

8.8     Maximum Time Available for Response and Correction

8.9     Priority Determination Grid

8.10   Critical Priority

8.11   Alarms Specified by Process Hazard Analysis (PHA) Teams

8.12   Safety Interlock Level (SIL) Alarms

8.13   Diagnostic Alarms and Other Special Priorities

8.14   Alarm Setpoint Determination

8.15   Alarm Documentation

8.16   The Master Alarm Database

8.17   Alarm Classification

8.18   After the D&R – Implementation of Changes

8.19   Training

8.20   D&R – Expected Duration, Costs, and Helpful Advice

8.21   Helpful Hints for Conducting a D&R

8.22   Common Action Items from a D&R

8.23   Staged Approaches to Alarm Rationalization

8.24   Staged Alarm Rationalization Methodology Details

8.25   Expected Results from Staged Rationalization

Step 5: Alarm Auditing and Enforcement

9.1     The Entropy of an Alarm System

9.2     Control System Alarm Change

9.3     The Dangers of Alarm Suppression

9.4     Alarm Auditing and Enforcement (A&E)

Step 6: Advanced, Real-Time Alarm Solutions

10.1     Application of Advanced Alarm Management Solutions

10.2     Alarm Shelving

10.3     State-Based Alarming

10.4     Guidelines for Implementation of State-Based Alarming

10.5     State Detection Logic

10.6     Recommendations for Shutdown State Alarm Settings

10.7     Alarm Flood Suppression

10.8     Design of a Flood Suppression Strategy

10.9     Operator-Adjustable Alarms

10.10   Operator Alert Systems

10.11   Alarm Handling Software – Write Your Own?

Step 7: The Control and Maintenance of Alarm System Improvements

11.1     Management of Change of Alarm Systems

11.2     Advanced Alarm Monitoring and KPIs

11.3     Annual Audits

Understanding and Applying ANSI/ISA-18.2: Management of Alarms for the Process Industries

12.1     Purpose and Scope

12.2     Does ISA-18.2 Apply to You?

12.3     Definitions in ISA 18.2

12.4     Alarm State Transitions

12.5     The Alarm Management Life Cycle

12.6     Life Cycle Stages vs. Activities

12.7     Seven Steps vs. Life Cycle Stages

12.8     The Alarm Philosophy Life Cycle Stage

12.9     Alarm Classification

12.10   Highly Managed Alarms

12.11   The Alarm System Requirements Specification (ASRS)

12.12   The Alarm Identification Life Cycle Stage

12.13   The Alarm Rationalization Life Cycle Stage

12.14   The Basic Alarm Design Life Cycle Stage

12.15   Human-Machine Interface Design for Alarm Systems

12.16   Enhanced and Advanced Alarm Methods

12.17   The Implementation Life Cycle Stage

12.18   The Operation Life Cycle Stage

12.19   The Maintenance Life Cycle Stage

12.20   The Monitoring and Assessment Life Cycle Stage

12.21   The Management of Change Life Cycle Stage

12.22   The Audit Life Cycle Stage

12.23   Summary

The Future of Alarm Management

13.1     A Grim Reminder

13.2     A Conversation

13.3     A Knowledge Worker

13.4     A Vision for 2020

13.5     Transforming the Role of the Knowledge Worker Operator

13.6     Integrated HMI

13.7     Robust Basic Process Controls

13.8     Comprehensive Alarm Management

13.9     Automation and Information Systems Integrity

13.10   Safe Production Knowledge Retention and Decision Support

13.11   Summary

The Death of the Lightbox

A1.1     Advantages of a Lightbox

A1.2     Observations

A1.3     Loss of View

A1.4     Alarm Configuration Security

A1.5     Disadvantages of a Lightbox

A1.6     Proper Implementation of a Lightbox with a DCS

A1.7     The Death of a Lightbox

A1.8     Lightbox Rationales

A1.9     What About ISA 18.2 and Lightboxes?

A1.10     Conclusion

Alarm Data Types

A2.1     Alarm Attribute Changes

A2.2     Records of Incremental Changes

A2.3     Records of Bulk Changes

A2.4     Methodologies for Obtaining Alarm Data

A2.5     Operator Change Events

A2.6     Printer Port or Printer Emulation

A2.7     System Databases or Files

A2.8     OPC

A2.9     Custom Software

An Example Alarm Philosophy

Operator Effectiveness

A4.1     The Fundamentals of Operator Effectiveness

A4.2     High Performance HMI (Operator Graphics)

A4.3     Graphics Principles

A4.4     Use a Proper Hierarchy of Graphics

A4.5     Level 1 – Process Area Overview Displays

A4.6     Level 2 – Process Unit Control Displays

A4.7     Level 3 – Process Unit Detail and Support Displays

A4.8     Level 4 – Process Unit Support and Diagnostic Display

A4.9     Trends Are Essential

A4.10   Navigation and Functionality

A4.11   Hardwired Switches

A4.12   Seven Steps for Creating a High Performance HMI

A4.13   Control Loop Performance Optimization

A4.14   Benefits of Robust Controls

A4.15   Seven Steps for Optimizing Control System Performance

References

Index of Terms

ILLUSTRATIONS

Figure 2-1: A Control Wall with Alarm Lightboxes at the Top

Figure 2-2: Configured Alarms per Operator Has Increased Exponentially

Figure 2-3: Alarm Rates Commonly Exceed Operator Handling Capability

Figure 2-4: One of the Worst Alarm Designs in History

Figure 3-1: Process Upsets and Profitability

Figure 3-2: Process Disturbances and Layers of Protection

Figure 3-3: An Alarm-Related Process Upset

Figure 4-1: Multiply Coded Alarm Indication Elements

Figure 5-1: Spare Pumps – Alarming the Wrong Way

Figure 5-2: Spare Pumps – Alarming a Better Way

Figure 5-3: Interlock Valve Closure Alarms

Figure 5-4: Four Priorities and Diagnostic Alarms

Figure 5-5: Three Priorities and Diagnostic Alarms

Figure 6-1: Recommended Alarm System Key Performance Indicators

Figure 6-2: An Example Alarms per Day Graph

Figure 6-3: Example Graph of Alarms per 10 Minutes

Figure 6-4: Example Graph of Alarm Floods

Figure 6-5: Example Table of Alarm Floods

Figure 6-6: Alarms per Day for a Pretty Good Week

Figure 6-7: Alarms per 10 Minutes for a Pretty Good Week

Figure 6-8: Alarm Flood Breakdown

Figure 6-9: Alarm Counts Exceeding 10 in 10 Minutes

Figure 6-10: An Example of a Top 10 Most Frequent Alarms Chart

Figure 6-11: Example Table of Top Twenty Most Frequent Alarms:

Figure 6-12: Best Practice vs. Actual Alarm Configuration and Alarm Occurrence Priority Distribution

Figure 6-13: Effect of Chattering Alarms on Daily Alarm Rate

Figure 6-14: Top 10 Chattering Alarms

Figure 6-15: Chattering Alarm Contribution to Overall Alarm rate

Figure 6-16: Alarms per Day With and Without Chattering Alarms

Figure 6-17: Alarms by Type

Figure 6-18: Alarm Suppression

Figure 6-19: Table of Stale Alarms

Figure 6-20: Example Table of System Changes Needing MOC Control

Figure 6-21: Alarm System Performance Levels

Figure 6-22: Example Table of Controller Mode Changes in One Week

Figure 6-23: Chart of Operator Controller Changes per 10 Minutes

Figure 7-1: Improvement Amounts from Alarm Bad Actor Resolution

Figure 7-2: Deadband and On-off Control

Figure 7-3: Deadband and Alarms

Figure 7-4: Deadband Settings Based on Sensor Type

Figure 7-5: Filter Time Constants Based on Sensor Type

Figure 7-6: Chattering and Fleeting Alarm Durations and Intervals

Figure 7-7: Alarm Delay Time Analysis Graph

Figure 7-8: ON-Delay (Duration) Histogram Percentage Determination

Figure 7-9: Delay Time Alarm Reduction Table

Figure 7-10: Recommended Delay Times Based on Signal Type

Figure 8-1: Areas of Impact and Severity of Consequences Grid

Figure 8-2: Electrical Generation Impacts

Figure 8-3: Maximum Time Available for Response and Correction Grid

Figure 8-4: Severity of Consequences and Time to Respond Grid for Alarm Priority Determination

Figure 8-5: Process History and Alarm Setpoints

Figure 9-1: Alarm System Uncontrolled Change

Figure 9-2: Alarm Suppression in Significant Amounts

Figure 10-2: State-Based Alarming

Figure 10-3: Proper Alarm Settings for the Low Energy State

Figure 12-1: The ISA-18.2 Life Cycle Diagram

Figure 13-1: Transforming the Role of the Operator

Figure A1-1: A Typical Lightbox

Figure A1-2: Which is Better?

Figure A4-1: A Typical, Low-Performance Process Control Graphic Containing Many Poor Practices

Figure A4-2: A High Performance Level 2 Graphic

Figure A4-3: Factors Affecting Control Loop Performance

Figure A4-4: Loop Performance Assessment Methodology

Figure A4-5: The Seven Step Control Improvement Process

Foreword

Unintended Consequences

by

Jim Pinto

You are reading an important book, about an important problem and its solution.

Technology has often advanced faster than our ability to use it effectively. We embrace the newest, shiniest technical toy and only later realize the unintended consequences. The manufacturers of our process automation technology innovate furiously, giving the end user thousands of choices and capabilities. It is in the use (and misuse) of these capabilities that we achieve both spectacular success and spectacular failure! And such failure is often due to the adoption of technology in the absence of wise guidelines for its effective use. This book is written to provide exactly that – wise and experienced guidance about a very problematic area of technology, the modern industrial process alarm system.

The advance of automation technology has enabled improved processing and system optimization for complex mixes of raw materials, operating environments, and process methodology adaptations. Global competitiveness and demand has required increased production using less energy, with less waste. Effective alarm system design and management are key elements in meeting these challenges.

Many alarm systems have been implemented without any proper guidance, and what we now know to be very poor practices were implemented and continued. Vastly over-alarmed systems producing thousands of alarms per day became common. Poorly performing alarm systems have been cited as specific contributing factors to major accidents and losses.

That's where this book is useful. It is written by individuals with vast experience in the different plants, processes, and environments requiring effective alarm management. It is filled with good examples and explanations of procedures, with practical lists and tips on how one should proceed. It is based on hundreds of successful projects.

This book is practical and instructive, written from a vendor-neutral standpoint making it valuable to suppliers, integrators, and end-users alike. It is a book for managers, process engineers, and operators; an essential textbook to keep around and refer to regularly. I particularly enjoy the practical quotations and real-world examples. Since the first edition of this book was published, alarm management has advanced considerably and remains a high-profile topic. Three years worth of additional experience has provided new topics and insights in the content and examples in this second edition.

The Future

Process automation technology will continue to rapidly advance in both capability and complexity. The control system must become self-correcting and self-optimizing – which means the system must adapt heuristically to reduce, not increase, the need for operator intervention.

True process effectiveness will not result from training operators to manage increasingly complex systems; it will come from training the system to incorporate the knowledge of the operator. This is one way the aging workforce problem and the near term retirement of most experienced plant personnel will be addressed.

The authors have performed a real service to the process industries by the publication of this second edition.

Jim Pinto

San Diego, CA.

October, 2009

Jim Pinto is founder (and formerly President and CEO) of Action Instruments. He is a technology futurist, venture capitalist, speaker, industrial automation commentator, analyst, and consultant. He has authored two books and writes for several webzines, journals, and magazines. Jim serves as an international consultant in strategic business planning, marketing, sales channel development, technology planning, and acquisition strategy. He is a member of the Association of Professional Futurists and the World Future Society.

The Alarm Management Handbook

A Comprehensive Guide

Second Edition

Practical and proven methods

to optimize the performance

of alarm management systems

Introduction

…

There are more things to alarm us than to harm us, and we suffer more often in apprehension than reality.

- Lucius Annaeus Seneca

…

Why We Wrote the Second Edition

In 2006, we published the original edition of The Alarm Management Handbook. It was then republished by the ISA (with minor changes) as Alarm Management: Seven Effective Methods for Optimum Performance. The response to these books has been amazing. Thousands of copies have been sold. In some cases, single companies bought dozens of copies for their multiple sites and engineering organizations. We've had positive feedback from hundreds of readers.

The purpose of the original book was to capture in one volume the current body of best practices knowledge for improving and optimizing the performance of a modern alarm management system. The book focused on practical advice, strategies, and techniques. In 2006, there was no such alarm management reference book with such a practical focus.

The past three years have seen much growth and improvement in the alarm management landscape.

Alarm Management has consistently remained as a high profile topic at technical symposia.

Control systems manufacturers are beginning to understand the problems inherent in their system designs, and are making improvements.

System implementers have begun to realize that rule-of-thumb methods of alarm configuration will cause big problems for the system owner, and are adopting improved methods.

Third party companies have continued to lead in technical innovations and advanced alarming solutions.

Hundreds more successful implementations of alarm management principles have taken place, providing additional terabytes of data which continue to confirm the validity of the principles in this book.

The electric power generation industry has begun widespread and concerted efforts in alarm management, accompanied by an alarm management recommended practice document published by the Electric Power Research Institute (EPRI), a document co-authored by the authors of this book.

The American Petroleum Institute (API) is working on a recommended alarm management practice for pipelines (RP-1167), hopefully to be published in 2010.

The ISA has (finally!) completed and published a standard on alarm management, an effort underway since 2003. This second edition contains information on this important development.

Given these factors, it makes sense to update the content within this book, and to provide additional guidance on some topics based upon the latest data and experiences.

This second edition, like the original work, will remain an intentionally different kind of book than you usually find in engineering circles. It is based on hundreds of person-years of extensive experience working with industrial control and alarm systems in almost every industry. All manner of practices make up the background information used in producing this book – the good and the bad, the best and the worst. The basis includes a working knowledge of the guidelines, standards, articles, reference works, and other materials on the subject, along with knowledge and experience obtained from hundreds of alarm management improvement projects.

Breakthrough results have been achieved by following the principles contained in this book. The principles herein can also enable new systems to be initially configured correctly, and not require expensive re-engineering after problems later become apparent.

In this book, you will find actual examples of good practices and poor practices. The various problems of alarm systems are covered with precise guidance on how they come about and how to effectively correct them. We know operating companies are limited by time, money, and resources. We will not advocate academic, theoretical, or impractical approaches to the problems. Instead, in this book you will find fact-based, field proven, straightforward, and practical solutions.

Changes and Additions in the Second Edition

This second addition has an additional fifty pages of information and many new figures. Questions and comments from readers of the original edition helped shape the new content. Much of the additional text provides more thorough discussion of specific topics, and almost every section in the book has been revised. There are also some completely new sections and chapters as well.

The major changes and additions include:

Where are we now – 4 years after the first edition

Details and impact of the new standard ANSI/ISA-18.2-2009 Management of Alarm Systems for the Process Industries

Changes in the regulatory environment

Additional discussion on the human factors issues associated with alarm analysis

Additional information about alarm management and batch and discrete manufacturing processes

Additional information on alarm bad actor resolution

Additional information on diagnostic alarms

Alarm classification

Improvements in alarm rationalization techniques

A thorough examination of staged approaches to alarm rationalization

Additional information on advanced real time alarming techniques

An update on the future of alarm management

An enhanced appendix on alarm philosophy documents

An updated appendix on both High Performance HMI concepts (proper operator graphics) and control loop optimization methods

Is this book for you?

This book specifically targets alarm management related to modern Distributed Control Systems (DCSs). This designation includes SCADA systems (Supervisory Control and Data Acquisition). These flexible and capable systems are used throughout various industries, including oil and gas, refining, chemical, petrochemical, pulp and paper, pharmaceuticals, power generation, minerals processing, discrete manufacturing, and others. Both DCS and SCADA types of control systems have identical concerns and solutions to alarm management issues. In this book, use of the term DCS includes SCADA systems.

The