Managing Change In An Open Process Control Environment

Managing Change In An Open Process Control Environment 2

Table of Contents
Introduction ................................................................................................................................................................................3

Background .............................................................................................................................................................................3/4

Complexity Issues ......................................................................................................................................................................5

Administrative Challenges .....................................................................................................................................................5/6

Best Practices.............................................................................................................................................................................9

Outside Support .........................................................................................................................................................................9

Conclusion................................................................................................................................................................................10

References ................................................................................................................................................................................10
Managing Change In An Open Process Control Environment 3

In the process industries, plant automation was once executed by centralized, proprietary control systems. A single automation
supplier, employing a large team of systems experts, was responsible for maintaining the process control environment, updating
equipment as needed and restoring operation in the event of a failure or shutdown.

Modern process control strategies are transitioning from closed, proprietary platforms to networks of flexible, distributed databases and
open Human-Machine Interfaces (HMIs) where every user has a PC or workstation. However, this trend raises an important question:
Who manages increasingly complex, geographically dispersed open systems employing hardware and software from multiple
suppliers?

The following paper describes the changes (and their unintended consequences) brought about by today’s open technology
environment.

Introduction
In the process control world, end user requirements are constantly changing. Early needs were limited to supervising and controlling
process loops. More recently, functions such as advanced control, alarms, and interfacing with business systems have been added. In
this environment, users need to optimize their assets, adhere to environmental regulations, and document production processes. All
this put greater requirements on modern process control systems.

Today, automation system users demand the latest open, interoperable architectures providing high performance and highly available
controls. The systems must be highly flexible and able to control the entire plant, transfer information, and provide places to store
history and information. In addition, users want greater flexibility to modify their control system without shutting down production
processes, and the ability to network easily to existing and future control components.

Background
In the past, a Distributed Control System (DCS) or Programmable Logic Controller (PLC) would have been the only means to apply
control applications. Plants could network these older systems together, and then engineers would have to manually route each point,
costing a user time and money. On top of all that, considerable time was required to install and set up special cables and hardware to
connect the systems to work as one.

Users seek integrated systems. Industrial manufacturers now seek to remove the constraints of closed, proprietary control systems
and free up their plant's profit potential. There is a growing demand for an open automation infrastructure integrating installed assets
and co-existing with legacy systems—all while protecting valuable investments.

Instead of relying on costly high-end workstations and other proprietary control system hardware, end users want the freedom to
purchase less expensive commercial PCs and utilize open software applications based on the familiar and user-friendly Windows
operating environment.

In addition, industrial facilities require solutions integrating process control and information systems with both plant and corporate
business systems. Instead of an integrated information system, users are recognizing the value of a unified information and application
software system. A unified system reduces the problem of data exchange and synchronization and maintains information in a central
repository shared by all business applications.

Open standards gain acceptance. Systems connectivity or interoperability is becoming more and more of an issue for plant
automation. Many DCS suppliers have changed their system architectures (and their attitudes) from proprietary to open systems.
Having an open system means that automation vendors make their interface protocols available to other suppliers, who can write
interface protocols to these systems, if they desire to do so.
Managing Change In An Open Process Control Environment 4

Open systems built from commercially available off-the-shelf products, and conforming to well-defined standard specifications, are
becoming the norm in the industrial automation market. Products conforming to these standards are easily connected to each other,
allowing a system to be quickly assembled. Systems built from open components provide different benefits to end users, systems
integrators, and OEMs.

Open process control systems, unlike closed, proprietary systems, enable end users to choose a "best-in-class" solution from
competitive offerings at cost-effective prices. Users are not limited to a single vendor, enabling the addition of new devices or
applications to their system as they emerge on the market. They can incrementally update their system one product at a time, guarding
against obsolescence.

IT versus process control. The transition from proprietary to open technology has highlighted the differences between the plant IT and
process control departments—as well as the criticality of their respective systems. Indeed, many software-based solutions common to
IT organizations are unsuitable for use with process control systems.

IT typically is business driven and focused on enabling users to perform their business activities while maintaining the integrity of the
system and protecting the company’s intellectual property that resides within the system. Process control regards human and plant
safety as its primary considerations. As IT focuses on the central server, process control emphasizes the edge client. IT understands
the online environment, but process control requires real-time performance and continuous operation.

Unlike the typical business network, an industrial-strength Process Control Network (PCN) must deliver the same level of robust
operation with open technology that end users have come to expect in the plant automation environment. In process control
applications, components must work together seamlessly in the same system. The hardware must be more robust to endure vibration,
heat and other harsh conditions not present in the typical office location. The embedded software must be fault tolerant and able to
execute where many home PCs would fail. The control hardware and application software must provide the control types required for
plant wide automation.

Building a secure PCN requires close attention to certain aspects of network design. Technologies used, and the deployment of those
technologies, must tightly secure the network environment from internal and external attacks or breaches. Both proactive and reactive
protection methodologies must be implemented to ensure mission-critical communications run unaffected by security events. Plus,
centralized command and control of security policies must be realized for operational effectiveness

Evolving technology changes roles. In addition to the new communication systems and architecture supporting the evolving process
control environment, there is the changing requirement of operation of these systems. Without the comprehensive support of
proprietary DCS experts, the establishment of process IT departments are now required to manage the day-to-day administration of
open control systems and their associated network infrastructure.

Automation end users must now assume greater responsibility for engineering a PCN with the reliability, robustness, performance, and
security traditionally associated with process control systems delivering real-time, mission-critical information. Whereas control
engineers and operators once had a limited role in managing their DCS environment, they must now deal with many more parameters
affecting control system operation. The number of user-changeable system files, registry settings, and other accessible administration
files increases exponentially in an open system architecture.
Managing Change In An Open Process Control Environment 5

As process control environments converge into the overall architecture of the enterprise network systems, IT organizations will likely
have to take on the added responsibility of securing not only the traditional computing environments of the enterprise network, but also
the converged mission-critical data of the process control system. Access control technologies have become even more crucial with the
requirements of user and system communication in a process control setting. Proactive protection must include all converged systems,
threat response technologies must be effective in recognizing and mitigating potential dangerous events occurring anywhere in the
converged network, and remediation of vulnerable and untrusted systems must be safely administered on the network system without
risk to neighboring devices and systems.

Complexity Issues
As previously stated, there has been an almost universal adoption of open system architectures based on commercial operating
systems and networking components in new control systems developed over the past 10 years. This trend started with cost-effective,
PC-based Human-Machine Interfaces (HMIs), and engineering stations with optional use of Ethernet have now extended to almost
every layer of new control systems.

Open technology has also impacted control system engineering and configuration. While proprietary systems were almost 99 percent
engineered and configured in a design center and then packed in a box and shipped, open systems tend to have many more variations.
Since each user site has unique characteristics, each open platform configuration has to be specific to a given installation. As such,
many engineering practices that were once standardized must now be redeveloped in the field—adding to project complexity.

Despite the acceptance of open technology for process automation, important questions have been raised about data integrity and
consistency. Open standards are necessary but not sufficient to truly integrate information. Thus, the issue of data ownership must also
be addressed. Who owns process and business data, and how does this data spread throughout the organization, reliably and
consistently?

Greater vulnerability. The benefits of open systems technology are many, which have helped to drive its rapid adoption. Open system
architectures, however, bring new risks often not fully recognized by users. The older, more structured proprietary systems, while less
flexible, naturally applied a layer of protection that is no longer present unless specifically added or preserved in new systems.

Increased complexity also brings greater vulnerability and reduced availability of individual components within a plant control system.
The technology used for plant and business IT networks has to accommodate an ever-expanding catalog of requirements. The number
of hardware and software platforms or versions used is growing exponentially in today's enterprise.

Increased maintenance. Highly complex open systems with interdependent components evolve through the interaction of individual
components, as well as the use of system-wide data—an imposing challenge for availability and security. The effort required to
maintain an extensive multi-layered security infrastructure has increased by an order of magnitude in recent years. This represents an
additional workload often not sustained with in-house IT resources, leading to decreased availability and increased vulnerability.

Administrative Challenges
In the days of proprietary control systems, system administration requirements were limited and easily handled by the typical process
control engineer. Modern open systems allow for greater flexibility in hardware and software, but with that flexibility, there is a greater
need to manage software releases, hardware compatibility, and system integration.

In addition to the increased size and complexity of the open technology environment, the responsibility for change management now
resides with the end user. For example, the task of managing new hardware and software releases has shifted from the main
automation contractor to the plant engineering department. This transition of responsibility can consume a tremendous amount of time
focused on implementing and maintaining control equipment, whereas the value to the user is on improving process performance.
Managing Change In An Open Process Control Environment 6

Today’s process control departments are challenged to address areas requiring specialized IT skills—skills more closely aligned with
corporate networks. These areas include:

Firewall management. Due to an increasingly complex and vulnerable enterprise, plants must implement a secure firewall on the
perimeter of their PCN infrastructure. Any connection into the PCN is considered part of the perimeter. Often, these perimeters are not
well documented and some connections are neglected. All entry points into the control system LAN should be known and strictly
managed by a security policy.

The most common entry points include: vendor access (potential way for an attacker to access the system without having to penetrate
the firewall), corporate LAN connection (since the corporate LAN is connected to the Internet, this connection has the most potential for
giving an attacker access to the control system LAN), and communication lines (for example, RTU connections to the control system
passing through networking equipment on the business network).

Well-configured firewalls are critical to PCN security. Communications should be restricted to only what is necessary for system
functionality. Control system traffic should be monitored, and rules should be developed allowing only necessary access. Any
exceptions created in the firewall rule set should be as specific as possible, including host, protocol, and port information.

Proactive firewall maintenance (including in-depth log file analysis of intrusion attempts into the PCN) and troubleshooting addresses
software/firmware upgrades, configuration and change management, data backups, device resource maintenance (such as rules,
access lists, and filters), plus device and cable plant interface management for device fault resolution.

Security policies. Open networking and distributed computing systems provide organizations with convenient access to information
and resources. However, this convenience has also made it easier for hackers, competitors, and disgruntled employees to compromise
the computing environment. Security measures appropriate for data networks could be disastrous if a compromised process threatens
to damage productivity, capital assets and, possibly, human life.

In an open network architecture, threats are most commonly posed by virus attacks, worms, network spoofing, impersonation, denial of
service, eavesdropping, password cracking, data tampering, and packet modification. For more information on cyber security, go to the
Honeywell Process Solutions library on cyber security.

In addition to the introduction of new technologies into the process automation platform itself, there is the trend of widened visibility and
operational models. Many control environments are moving toward the inclusion of a secure interface with the business network. This
accommodates an operational model including involvement by offsite experts and vendors, but also opens the door to additional access
control security concerns. Careful consideration must be given to balance the benefits of a highly efficient operational model against the
potential risks that come with the expanded connectivity required to realize this model.

With the increasing dependencies of open-standards Ethernet and TCP/IP-based network systems in the process control environment,
it is now critical to address the security vulnerabilities common to these communication infrastructures. Leveraging the pervasive nature
of the network infrastructure can provide a valuable asset in the overall approach to securing critical infrastructure. The use of
authentication, access control, proactive protection, and dynamic response technology provides the best holistic approach to network
security in process control. Currently, most controllers and PLCs provide no authentication or access control, making a layered
approach to security even more important.
Managing Change In An Open Process Control Environment 7

Level 4 Comm flow

No Direct communications between L4 & L3

Level 3.5 L4 to L4
Terminal
OS Patch Level 3.5 Business
& Virus
Services Protection DMZ LAN Wireless DMZ
Data Srv Srv

Very Limited
Syn eServer
Svr PHDS
Switch Pair

L3.5 to L4
Limited
L3.5 to
L3.5
NON-FTE redundancy

Very Limited L3
Domain
BCK Controller &
DVM PHD/S Svr IAS Switch Pair Level 3
Advanced

to L3.5
Control

or L2
L3 to L3
Optional Router depending on PCN
NON-FTE redundancy complexity (may connect directly to FW)

Very Limited

No communications between L1 & L3

L2 to L3
PKS

Limited

L2 to L3.5
Svrs ACE ESF ESC DC Switch Pair
Level 2 ES- ESV PHD
T T
Supervisory
Control
L2 to L2

FTE LCN

L2 to L1
Limited
FTE Control Switch Pair
Firewall Level 1 FSC
PM
NIM
Pair Process Family
L1 to L1
C200 Control

or L4
C300 Controller
Controller

Figure 1: Layered approach to control and security of the PCN

Open systems users can mitigate their information security risks by implementing an overall security strategy for the corporate and
plant enterprise. The most important step in this effort is developing an effective plan of attack. Companies should write policies and
procedures aligning with corporate security goals, and perform thorough risk assessments identifying plant and business assets, define
potential threats, and spell out vulnerabilities across the enterprise.

Virus protection. It is extremely important for plant IT organizations to keep their antivirus software up to date. New viruses and other
malware are cropping up all the time, and manufacturers of antivirus software are working constantly to identify and eradicate new
threats. Keep in mind out-of-date antivirus software is almost as bad as having no antivirus software at all.

As we all know, anti-virus solutions can only detect those viruses they “know” about. This information is usually referred to as virus
definition/signature/pattern, which must be updated on a regular basis. Most anti-virus solutions update their definition/signature/pattern
automatically via the Internet at regular intervals. However, it is important that these anti-virus updates be tested and validated by the
control system vendor to ensure full compatible prior to installation.

Best practices should be employed to keep PCN systems up-to-date with anti-virus definition patches (including general virus protection
definitions and special security update definitions), using an automated, yet controlled process. It is also desirable for the PCN to have
its own anti-virus update server located on a PCN Demilitarized Zone (DMZ).

Patch management. Operating System (OS) patches repair vulnerabilities that could allow an attacker to exploit the control system
architecture. The importance to system security of keeping OS patches up-to-date cannot be over-emphasized. Nevertheless, patching
control system software can present unique challenges. Among the factors to consider are system functionality, security benefit, and
timeliness.
Managing Change In An Open Process Control Environment 8

Applying security fixes is part of the task of software management, as IT managers know all too well. There are regular security patches
issued from Microsoft and other Windows-related vendors. (Some of these releases may actually encompass several patches,
representing a much greater number of actual patches.)

IT specialists must verify the OS patches will not affect the functionality of process control applications before implementing the
patches. Since process automation systems are normally in continuous operation, and thus rarely reboot, there are fewer opportunities
to download and install patches. Because direct communication between the process control and the enterprise networks is not
considered a best practice, it is preferable for the PCN to have its own OS patch management system. A best practice of vendors is to
validate all OS patches to ensure compatibility with the control operating system prior to release to the customer.

Managing the installation of patches is essential to ensure timely, accurate patching is done in a way that avoids or minimizes downtime
to key systems. For small enough process IT organizations, the updating tools built into their operating systems and applications may
be sufficient to keep up with release patches—if IT is willing to trust the safety of the patches and if there's enough Internet bandwidth
to let each computer download its own copy of each patch. However, for organizations of greater size or IT complexity, a
comprehensive OS patch management program must be initiated.

Data recovery. Today’s competitive business environment demands plants and mills remain online 24 hours a day, seven days a
week. Loss or corruption of process data can have a negative impact on a manufacturer’s ability to meet business demands.

The best defense against this situation is an effective data recovery strategy. However, before addressing data backup, it is important
to assess the impact of data loss, corruption, and availability on the various areas of the process. A data recovery assessment can be
used to identify existing data recovery procedures and tools, and then review these procedures relative to changes or additions in the
process control system.

The data recovery assessment should identify the end user’s current backup retention policy, including the offline retention policy, with
consideration for timing issues related to availability of off-line/off-site backups. The assessment should also identify data recovery
requirements for the process control open nodes based on the function of each node, providing type of backup required and backup
frequency for each node.

Risk assessment tools based on Six Sigma principles can determine the operational impact from the loss of each node. This risk
assessment helps to identify exposure to data loss and to prioritize vulnerabilities. It also pinpoints external threats to data and its
recovery.

Interface management. Traditionally, each software or application developer in the process control market was required to write a
custom interface, or server/driver, to exchange data with hardware field devices. The industry addressed this situation by developing
open standards such as OPC defining a common, high performance interface easily reused by HMI, SCADA, control, and custom
applications.

The OPC specification is a non-proprietary technical specification defining a set of standard interfaces based upon Microsoft's Object
Linking and Embedding (OLE) technology for Process Control. The application of the OPC standard interface enables interoperability
between automation and control applications, field systems and devices, and business/office applications.

Despite the advantages of interoperability made possible by OPC, end users are still challenged with managing software interfaces to
ensure their control system can talk to a wide variety of third-party devices installed on the plant floor. Communication failures resulting
from patches and upgrades can occur throughout the control system infrastructure.
Managing Change In An Open Process Control Environment 9

The solution to this problem is not easy. Typically, it involves reducing the number of OPC devices and, to some degree, limiting the
openness of the control system architecture. End users should strictly control the number of upgrades on devices and software
applications, and always test and validate upgrades offline before putting equipment back into service.

Best Practices
Industrial end users should strive for a culture embracing change management within their IT organization—overall, 80 percent of
system outages are due to change. Process plants need effective methodologies for managing configuration databases, system
capacity and availability, IT service levels, software releases, backups, and other key aspects of network operation and maintenance.

Many plants taking a proactive approach to managing their process IT system have adopted best practices based on the Information
Technology Infrastructure Library (ITIL), a comprehensive set of standards developed in 1989 by Great Britain's Office of Government
Commerce. ITIL is a widely accepted approach to IT service management providing guidelines drawn from both the public and private
sectors internationally. It is supported by a comprehensive qualifications scheme, accredited training organizations, and implementation
and assessment tools.

The benefits of implementing globally consistent ITIL-based processes include:

• Improved availability, reliability, and security of IT services

• Increased IT project delivery efficiencies

• Reduced total cost of ownership of IT infrastructure assets and applications

• Improved resource utilization, including decreased levels of rework and elimination of redundant activities

• Provisioning of services that meet business, customer and user demands, with justifiable costs of service quality

• More effective third-party relationships and contracts

Outside Support
Today’s aging workforce is a trend process industry operations must address now. U.S. industries have already witnessed the average
age of their workforce increase to its current level of 48 years of age. About 40 percent of senior engineers and supervisors in industrial
plants will be eligible to retire in 2009. The loss of expertise in most plants is exacerbated by the lack of new recruits entering the field.

Faced with the retirement of a large share of the industrial workforce, and the resulting shortage of skilled labor, plants need solutions
optimizing their process control infrastructure—and at the same time, taking the pressure off of on-site resources. This includes
effective measures for increasing open systems uptime and avoiding production losses.

To help customers cope with the complex open technology environment, major automation suppliers such as Honeywell Process
Solutions have developed service offerings focused on the assessment, design, and implementation of process control networks. Open
system services address important issues ranging from network administration and security, to data recovery, software management,
and performance management. They are a cost-effective alternative to maintaining an in-house process IT capability, and can help
keep process control networks running in a secure environment.

Remote PCN monitoring and management services are particularly beneficial for end users with limited internal IT resources. These
services provide qualified outside IT specialists who monitor the availability of network infrastructure devices and connected systems,
and establish a secure remote access facility for mission-critical systems.

Remote PCN monitoring keeps tabs on bandwidth reductions, data throughput issues, error rates, and other problems threatening to
impact production. It also ensures proactive threshold alarming on network devices, systems, and applications. Remote PCN
Managing Change In An Open Process Control Environment 10

connectivity management provides the means to deliver of both pro-active and reactive services, and is an effective tool for managing
firewalls, patches, and anti-virus updates.

One of the benefits of remote infrastructure monitoring is in the centralization of all critical events and trouble alerts through a common
notification service. In addition, while users focus on their core business, the automation supplier keeps tabs on the network. If a device
becomes unavailable, an alarm notification event is triggered. Once an alarm is determined to be abnormal, plant personnel or the
responsible vendor is notified.

Conclusion
The move away from proprietary control platforms to open systems has provided process plants with significant performance
improvements and cost savings. However, this transition has also posed major information technology challenges. End users can no
longer rely on their automation supplier for 24/7 support to keep the proprietary DCS environment solid and secure. In many cases,
plant engineers and operators are now responsible for increasingly complex open automation architectures—and the specialized IT
support functions that go along with them. These challenges can be readily overcome by a combination of assessing vulnerabilities,
designing and implementing secure, effective networks and through the ongoing management of these systems by teaming with the
automation supplier to provide best practices and knowledgeable resources.

References
1. ERICKSON, K. and HEDRICK, J., “Plantwide Process Control,” Wiley-Interscience, (1999).

2. “Mitigations for Security Vulnerabilities Found in Control System Networks,” ISA, 16th Annual Joint ISA POWID/EPRI Controls
and Instrumentation Conference, (2006).

3. BOED, V., “Controls and Automation for Facilities Managers,” CRC Press, (1996).

4. WATANABE, E. and FERNANDES, J.M., “Open Should Mean Open,” InTech, (Aug. 2006).

For More Information

To learn more about Honeywell’s Open System
Services, visit our website www.honeywell.com/ps or
contact your Honeywell account manager.

Automation & Control Solutions

Process Solutions
Honeywell
2500 W. Union Hills Dr.
WP-08-04-ENG
Phoenix, AZ 85027 April 2008
Printed in USA
Tel: 877.466.3993 or 602.313.6665 © 2008 Honeywell International Inc.
www.honeywell.com/ps