OVERVIEW OF THE

INTEGRATED
CONTROL
SYSTEM
ESS-0297798

2018-06-01
TABLE OF CONTENT Page

1. ICS IN 30 SECONDS 3

2. INTRODUCTION 4

3. ICS ARCHITECTURE 6

4. ICS TECHNOLOGIES 8

4.1. Software 8

4.2. Hardware 10

4.3. Networks and Infrastructure 12

4.4. EPICS 7 13

5. ICS FUNCTIONS 14

5.1. Overall Control System Function 14

5.2. Controls Configuration Management 15

5.3. Machine Protection 16

5.4. Main Control Room and Experiment Operation Consoles 17

5.5. Personnel Safety Systems 19

6. ICS PRINCIPLES 20

7. ICS ORGANISATION 21
7.1. Line Organisation 21

7.2.. ICS Project Management 23

7.3. Construction Project Organisation 24

7.4. ICS Information Management 25

7.5. Financial Key Figures 25

8. ICS VISION, MISSION AND VALUES 26

9. REFERENCES 27

10. DOCUMENT REVISION HISTORY 28

2
1. ICS IN 30 SECONDS
The Integrated Control System Division (ICS division) is an organisational unit responsible for the control
systems within the ESS facility including control systems for accelerator, target, neutron scattering systems
and conventional facilities. ICS’ vision is to see ESS operated efficiently, reliably and safely, with a control
system that everyone loves. The ICS division shall provide and maintain world-class and cost-efficient control,
protection and safety systems and services for the ESS facility. The division shall develop competence and
innovative solutions that can be shared in the community through open processes.

3
2. INTRODUCTION
Control Systems at the European Spallation Source

The neutron science experiments at ESS and the operation of neutron-producing facilities (accelerator, target)
rely heavily on a controls and information technology infrastructure. Many complex physical processes have
to be concerted, such as proton beam acceleration, neutron beam production, and the characterisation and
conditioning of experimental material samples. These processes depend on other enabling processes, such as
magnetic field generation, equipment cooling, motion control, supply of electrical power and highly sophisticated
measurements for their characterisation. The related equipment (electromagnets, neutron detectors, cooling
systems, etc.) needs to be controlled and monitored by operating staff, engineers and researchers. This is
achieved through many different and distributed but interconnected control systems.

The physical processes and system behaviours throughout ESS, from the ion source to neutron scattering
characterisation systems, are characterised by a high degree of interdependencies. The operating staff,
engineers and scientists need tools for the comprehensive supervision, analysis and optimisation of all
involved systems and processes. This is realised by a homogeneous data communication layer, which integrates
all technical systems that participate in neutron production and experiments at ESS. This communication layer
enables all involved systems to exchange data and provides the necessary tools for the analysis of information
acquired from these systems. The ensemble of these control and communication systems is commonly referred
to as the ESS’s Integrated Control System (ICS).

The creation of the ESS Integrated Control System requires a team of specialists in many fields, such as
industrial automation, software engineering, computer science, safety engineering, network infrastructure
design, electrical engineering, accelerator technology, neutron scattering technology and even natural sciences
domains. Within ESS, the ICS division is the organisational unit that is responsible for the development,
operation and maintenance of the Integrated Control System over the ESS life time. The ICS division covers the
core competencies required for this mission and establishes the required line organisation.

The ICS division develops a consistent and sustainable architecture for the control systems and related networks
and services deployed in the ESS facility. For this, the ICS division engages in ESS in-house standardisation
of controls and computing technologies, which range from commercial off-the-shelf products to highly
customised devices or even novel developments. During the ESS construction phase, the ICS division executes
the ICS construction project, which delivers the control system scope for the ESS greenfield facility.

Note: The abbreviation ”ICS” is in practice often used for three different purposes:

•  Technical: the control and communication systems at ESS,

•  Organisational: the ICS division,

•  Managerial: the scope of the time-limited ICS project during the ESS construction phase.

The outline of involved parties, science and technical domains indicate the diversity of ESS internal and external
stakeholders who the ICS division interacts with. “Figure 1: The ICS Division Stakeholder Overview” on page
5 gives an overview on the stakeholder groups that the ICS division often works with.

4
The outline of involved parties, science and technical domains indicate the diversity of ESS internal and external
stakeholders who the ICS division interacts with. Figure 1 gives an overview on the stakeholder groups that the
ICS division often works with.

Figure 1: The ICS division stakeholder overview

The complexity of stakeholder involvement in the control systems can be illustrated by looking at a process control system
as an example. Such a process control system may have all of the following stakeholders

•  one or several groups from other technical ESS divisions as owners of the controlled equipment,

•  ESS operations teams, with interests in the technical support for executing operational tasks and procedures
(such as switching on the ion source),

•  one or several ICS groups, as owners or providers of control systems or control system infrastructure,

•  external partners,

•  in-kind contributors,

•  commercial suppliers,

•  other central ESS functions such as the ESS Environment, Safety and Health division,

•  ESS end-users, who conduct neutron science experiments.

5
3. ICS ARCHITECTURE
The ‘Integrated Control System’ of the ESS is an ensemble of
interconnected, distributed control and computing systems for
different control domains and purposes. Some of the most prominent
systems and infrastructures are described in the following.

Facility Supervision and Integration control systems also control equipment and related
physical phenomena, but usually in a more accelerator
ESS uses the EPICS family of distributed control and research technology specific context, such as
system software [22] to realise a homogenous facility proton beam diagnostics, experimental sample
integration layer for all ESS systems relevant to conditioning and neutron detector systems. Device
neutron production and usage. This communication control systems are often realised as customised
layer enables all involved systems to exchange high performance control systems. See section 4.2 for
data, and provides operators, engineers, etc. with more technology information.
the necessary tools for supervising, analysing and
controlling these systems. See section 4.1 and 4.4
for more information.
Personnel Safety Systems
Providing and assuring safe conditions for personnel
Timing System is required to operate ESS. The ICS division provides
Personnel Safety Systems (PSS) to identify, analyse,
The ESS has a timing system [14] that provides and mitigate against the facility’s prompt ionising
high-precision synchronisation of the distributed radiation hazards, as well as other hazards such as high
control systems in order to achieve coordinated voltage, magnetic fields, lasers, motion controls and
operation of the accelerator, target and neutron oxygen depletion. See section 5.5 for more information.
instruments. The timing system is synchronised
to the accelerator power sources. It broadcasts
operational data, time stamps and trigger events to Software Services
a large number of receivers simultaneously with a The ICS division provides a set of software services to
deterministic nano-second accuracy. ESS developers, operators and users for operational
and maintenance purposes. This software provides
Machine Protection Systems graphic user interfaces for operation, machine data
analysis and archiving, equipment configuration,
ESS machine protection supports the operational calibration and maintenance. See sections 4.1 and 4.4
availability requirements for neutron production for more information.
by preventing equipment damage through
dedicated protection functions. In addition to
protecting equipment and avoiding unnecessary Control Room facilities
downtime, information on root causes of errors The ICS division provides the environment and
can be collected for further analysis and continuous equipment for ESS operators, scientists and engineers
system improvements. Machine protection can be for interaction with the ESS systems. The most
summarised as the P3 principle: prominent example is the ESS main control room. The
• Protect the equipment, main control room is the ESS’ central place where the
proton and neutron beam production is controlled
• Protect the beam, and supervised and related operational activities are
• Provide the evidence. executed. Additionally, the ICS division provides local
control rooms for the cryogenic systems and console
See section 5.3 for more information. workplaces for the neutron instruments. See section
5.4 for more information.
Process and device control systems
Process control systems interact with equipment
Networks and infrastructure
which enables or monitors physical processes. The ICS division provides technical data
Examples of systems that require process control are communication networks called the Technical
cooling systems, electromagnet systems or ventilation Network, centralised computing resources and data
systems, and the process control function is usually storage facilities needed for the operation of the ESS
realised with industrial automation technology. Device systems. See section 4.3 for more information.

6
Figure 2: ICS system distribution overview. This schematic indicates roughly the facility-spanning
character of the networks for EPICS, Machine Protection, the Timing System and the Technical
Network backbone, contrasting the more confined character of process and devices control
systems and Personnel Safety Systems.
Figure 2 indicates the on-site distribution of the various systems. While these systems are often independent,
they are also interconnected.

Layer Architecture of the ESS Control Systems

Human Users
engineers, operators,
scientists, technicians, ...
display ESS facility data , ESS Main Control Room Experiment workstations other human interfaces
task execution
Applications
GUIs for data analysis,
processing and archiving services, System Equipmet Physics Config. Data on PSS NICOS,
system configuration, ... GUIs GUIs Apps Tools Demand GUIs DMSC
ESS facility
data exchange EPICS
High Performance
Facility Integration Data Network
runtime & historical data, EPICS EPICS EPICS EPICS EPICS Data on EPICS
complex control functions, IOC IOC IOC IOC IOC Demand Archiver
configuration databases Event Timing System
Generator
system specific Machine Protection
Fast Beam
data exchange Interlock System
Local System Control Personnel
process and device control, MicroTCA Safety Systems
Ether MicroTCA PSS
high dependability, PLC PLC
CAT System System PLC
executes local control functions Fieldbus

local control,
run - time data
Signal Conditioning and Remote Device Device Remote Remote Device signals
Equipment Electronics I/O
Controller
Electronics Electronics
Controller
I/O I/O
analogue and digital signals device specific
signals
Physical Equipment
machinery, sensors, actuators Pump Flowmeter Motor Sensor Chopper Power Supply Switch Sensor Sensor

measure and adjust

Physical Processes H2O Flow He Flow Mech. Movement RF Proton Neutron

etc.
Temp Electricity

Figure 3: Layer architecture of the ESS control systems

Figure 3 is a schematic overview of control system layers and the relationships between physical processes,
equipment, control, communication and supervision systems and human users. Control systems in the layer
architecture view are presented in section 5.1.

7
4. ICS TECHNOLOGIES
The role of technical standards for the ESS control systems.

The control systems have to perform according to the ESS goals for enabling neutron science, but also provide
the flexibility to cope with modifications over the ESS life time. The need for enhancements, upgrades and
modifications have to be expected during commissioning and as experience is accumulated in operation.
Among the drivers for upgrades and modifications are general advancements in technology, and new research
interests that require novel experiment or instrument designs.

For the ESS control systems this means that their design goals are system specific trade-offs between
function, performance, cost-efficiency, safety, reliability and maintainability. A central principle for achieving
overall optimised control systems is to apply a limited, but versatile and powerful well-defined set of system
technologies. The ICS division defines, customises and maintains this technology portfolio, which is called
the “ICS standards”. Maintaining these standards is more than simply defining technologies – it also includes
providing the means to apply the ICS standards efficiently. Therefore, the ICS division also provides the needed
staff competences, enabling systems such as for code management or testing, work procedures, and maintains
good relationships with suppliers and collaborations with other external organisations. The ICS standard
portfolio is continuously evaluated and adjusted, based on evolving ESS needs as well as trends in the related
research and industry domains. In the following sections, an overview of the core technologies used by ICS is
given. An extensive description can be found in the ICS Handbook [13].

4.1. Software
ESS uses the EPICS family of distributed control system
software for the creation of a facility-wide data communication
layer, which integrates all technical systems that participate in
neutron production and experiments.

EPICS

A set of EPICS-based software technologies and services form the core of the EPICS environment. Communication
between the various process and device control systems, databases, etc. are realised by the “EPICS base”. The
EPICS base consists of the EPICS protocol, programming libraries and configuration conventions for a variety
of operating systems, programming languages and hardware platforms. These are used to implement EPICS
Input-Output-Controllers (IOC), which connect to equipment distributed over the ESS site. ESS operators,
engineers and scientists can use various graphical user interfaces (GUI), including synoptic system overviews,
engineering screens for devices, and machine settings management applications. Applications help users to
manage the system configurations, which is important for repeatable and efficient configuration of layered
machine settings and sequences. Configuration management applications can also help users to incrementally
develop complex system configurations through working with ‘snapshots’ of previous parameter configurations.

In addition to the EPICS base for equipment integration and management, the ICS division deploys the following
software components and services which are essential to the operation of the facility.

8
Archiving Service

The EPICS archiving service is the time-stamped machine history archive, which collects and stores data
from the IOCs, and provides tools for retrieving and analysing the collected data. The archiver application
provides analysis functions for the visualisation, correlation and export of the collected machine history. The
EPICS archiving service is primarily aimed at troubleshooting, machine performance analysis, operations and
maintenance support. Neutron experiment data acquisition is realised with dedicated systems tailored to the
individual instruments.

Alarm System

The EPICS based Alarm System informs users about conditions that may interrupt, or have already stopped,
neutron production or can otherwise be harmful to the facility or its components. The alarms help personnel
to diagnose causes of such conditions, and to take appropriate actions to prevent or mitigate their effects. The
alarm system is essential for reducing downtime and maximising availability of the ESS services.

Control System Studio

ESS uses Control System Studio as the standard tool for run-time interaction with the ESS control systems. CS-
Studio is closely associated with the EPICS family of open-source software and included in the ESS standard
software distribution. The ICS staff has a high level of expertise in using this tool and can create feature-rich
graphical user interfaces. System owners and engineers can also create customised GUI applications for their
systems easily through drag-and-drop functionality.

Electronic Logbook

Users can document decisions, actions and notable events and observations during operation in an electronic
logbook, which is also required by regulatory authorities. Integrated into the ESS control application suite, the
logbook is used to document operation events in various media formats, including text, pictures, screenshots
and hyperlinks, including links to ESS information systems.

Accelerator Machine Model

ESS uses the OpenXAL development environment as the framework for high level physics applications which
are needed to simulate beam trajectories of the proton beam. A machine model of the ESS accelerator allows
to translate the simulation results to machine settings and apply these to the equipment during operation.

Control Configuration Data Base

For management of the control systems, the ICS division uses the Control Configuration Data Base (CCDB)
environment, which includes closely related services and guidelines such as for ESS Naming [18]. The CCDB
environment is explained in more detail in section 5.2.

9
4.2. Hardware
The ICS division has adopted three standardised hardware
technologies for implementing the ESS control systems based
on performance [13].

These three hardware technologies are MicroTCA, EtherCAT and common industrial automation technology.
A specialised technology is used for the ESS Timing System. All systems based on these technologies are
integrated using EPICS.

10 MHz MicroTCA® is a modular, open standard for building high performance switched
fabric computer systems in a small form factor. Because of its modularity and
flexibility, MicroTCA is well-suited for industrial control and automation systems.
MicroTCA defines fully redundant redundant system configurations including
1 MHz power budgeting, hot-swap, complete component and system management that
allows failure detection and isolation. This greatly improves system reliability and
Digital Controls Platform availability.
100 kHz
10 kHz
EtherCAT is a modern high-speed industrial automation system standard
1 kHz which enables faster data interchange with deterministic timing and higher
processing power than traditional industrial automation systems. EtherCAT
systems at ESS are connected to the EPICS control layer.
100 Hz
EtherCAT

10 Hz

Slower signals are handled by industrial automation (PLC) for reliability and
1 Hz cost reasons. The standardised platform for ESS applications comes from
Siemens.

0.1 Hz Industrial automation (PLC)

Figure 4: Control technologies and performance

With these three levels of technology performance, the full range of required signal speed can be covered while
complying to the strict reliability and availability demands at ESS. The standardisation of these technologies
makes the implementation cost-efficient and maintenance relatively simple. A standard technology overview
is given in Figure 4.

10
MicroTCA

High performance signal processing and data acquisition functions are implemented on a custom-made
platform based on the modular MicroTCA standard. Typical systems in this category contain computers in the
MicroTCA form factor that can run EPICS environments. This is complemented by flexible and powerful data
acquisition and processing engines typically implemented in high density FPGA technology. The physical front
end to these systems is also modular, typically using the FMC form factor for interfacing electronics.

EtherCAT

For mid-range performance, industry standard EtherCAT systems are used to implement real time fieldbus
applications with a good price/performance ratio. The pulsed character of the ESS beam production requires
synchronisation and event-specific information in a significant number of use cases where EtherCAT technology
can be applied and full custom platform solutions would be too costly.

Industrial Automation Systems

The low end of the performance spectrum is handled with commercially available industrial automation systems
supplied by Siemens. This cost-effective solution also addresses ESS reliability and maintainability needs and
integrates well into EPICS.

Timing system technology

The ESS Timing System is based on technology from Micro Research Finland (MRF) who specialises in developing
precision timing systems for accelerators or similar large-scale scientific applications [14].

Facility integration using EPICS

The presented technologies are used to realise control systems and equipment integration as shown in the
“Local System Control” layer in Figure 3. To enable data exchange with the rest of the facility and centralised
software services such as the EPICS archiver, each of these local control systems is controlled by an EPICS
Input/Output Controller (IOC), shown the “Facility Integration” layer in Figure 3. These EPICS IOCs can be
realised through standard Industrial PCs, but the majority of IOCs are handled with virtualisation technology
as described in section 4.3.

11
4.3. Networks and Infrastructure
All sub-systems and components of the distributed ESS control
systems, human workplaces and data processing facilities are
interconnected by a set of data networks.

The Technical Network

The Technical Network is the major backbone data carrier for control system information exchange. It enables all EPICS
based information exchange between local control systems, local devices, operator consoles, control system computing
and data storage facilities. The technical network is built with approximately 300 network switches, routers and firewalls,
and about 235 km of copper and optical fibre cables. A site-wide, high performance, bi-directional optical fibre backbone
connects the various buildings, sections and experiments, where local copper networks branch out to local connection
points. The Technical Network is built with high availability, reliability and security in mind.

Figure 5: ESS physical network topology

Virtualisation and Container Technology

The ESS control systems demand a significant amount of computing resources for running EPICS and other software
services. A key strategy for realising high availability of these computing services is a heavy deployment of virtualisation
and container technology. This means that many of the EPICS IOCs and other software services are deployed on virtual
computing systems or isolated software processes, which are physically realised on high-availability, managed computer
clusters in centralised server rooms. This strategy allows for high availability by flexible resource allocation, load balancing,
a tightly controlled and protected environment and cost-efficient resource utilisation.

Timing System Network

The Timing System Network distributes timestamps, event triggers and selected operational data to all systems which
need high-performance synchronisation. It connects the central ESS timing system event generator with many local
event receivers using fibre optics technology [14].

12
4.4. EPICS 7

ICS engages in the development of the next generation of the

EPICS technology; EPICS 7. While this is primarily motivated
by the technical benefits for the equipment integration at ESS,
this engagement is also a contribution to the community which
made EPICS available.

EPICS is open source.

EPICS is an open-source software technology, widely used at accelerator and other large-scale research
facilities. It has been developed in this type of environment as a community effort for almost three decades.
This community effort includes the provision of the core libraries, hardware and operating-specific drivers,
modules and development tools. But equally important, the EPICS community engages in open sharing of
experiences (for example in workshops, conferences, email lists), practical help and staff visits for reviews and
training. In short, EPICS as technology and community lives and thrives on active engagement in its community.
By contributing to the development of EPICS 7, ESS can participate and strengthen the community and influence
the EPICS development roadmap.

3 + 4 =7
EPICS version 3 is the most widely deployed version today. To accommodate for the increasing demands of new
research facilities, some fundamental improvements were introduced in EPICS 4, including a new communication
protocol, new libraries and data types. However, EPICS 4 is not straightforward compatible with existing EPICS 3
control systems and software. EPICS 7 (from the combination of ‘3+4’) extends EPICS 3 with EPICS 4 functionality,
without breaking backward compatibility, and hence allows for full utilisation of the existing EPICS base.

The Experimental Physics and Industrial Control System

Main motivations for using EPICS 7 over EPICS 3 include:

•  Data handling and data types: Increasing relevance of 2-dimensional data, such as from area detectors,
cameras etc. motivates the support for more complex control data constructs.

•  Data transfer performance: Ever-increasing amounts of data drives optimisations of data transfer performance.
EPICS 7 implements several technology advancements for increased performance

•  Remote Procedure Calls: EPICS 7 introduces so called remote procedure calls which can use dynamic data
structures. This feature adds additional flexibility for IOC and application development and performance
optimisation.

EPICS 7 is already in limited use for operation at some accelerator facilities

13
5. ICS FUNCTIONS
5.1. Overall control system function
The low energy beam transport section (LEBT) is one of the
first sections in the ESS accelerator. It requires a representative
variety of control system functions and is used to illustrate the
application of ICS technologies as presented in chapter 4.

Figure 6 shows a schematic view of some of the control systems for the LEBT section. Operators in the main control
room and other users such as technicians use a set of GUIs to control, operate and maintain the LEBT. GUIs and tools
exchanging information with the ESS facility are shown on the “Applications” layer.

Layer Architecture of LEBT Control Systems

Human Users
engineers, operators,
scientists, technicians, ... ESS Main Control Room other human interfaces
display ESS facility data,
task execution
Applications
GUIs for data analysis,
processing and archiving services, System Equipment Physics Config.
system configuration, ... GUIs GUIs Apps Tools
ESS facility
data exchange EPICS
Infrastructure
Facility Integration BCM
IPC with
runtime & historical data, EPICS IOC EPICS IOC
complex control functions, Timing System
EVG, Master
configuration databases
Oscillator
system specific
Machine Protection
data exchange
FBIS

Local System Control

process and device control, Iris Motion BCM
high dependability,
EtherCAT ISrc_LEBT PLC
Control MicroTCA
executes local control functions Fieldbus
local control,
Signal Conditioning and run - time data
Chopper Pump PS ACCT-E-
Equipment Electronics Electronics Electronics Electronics RM Device signals
analogue and digital signals device specific
signals
Power supply for
Physical Equipment Temp Sensor Chopper Pump
Solenoids, Steerers
Iris ACCT
machinery, sensors, actuators
measure and adjust

Physical Processes Temp Proton H2O Flow Mech. Movement

Figure 6: Layer architecture of control systems in the LEBT section

14
The LEBT contains different kinds of equipment such as power supplies, pumps, and beam instrumentation
systems, forming the “Physical Equipment” layer. This equipment is connected to either equipment specific
electronics on the “Signal Conditioning and Equipment Electronics” layer, or directly to local control
systems, such as PLCs on the “Local System Control” layer. In any case, local system control is realised
with the ESS technology standards (MicroTCA, EtherCAT or Siemens PLC systems - see section 4.2). On
the “Facility Integration” layer, an Industrial PC (IPC) runs the EPICS IOC which handles all local control
systems and makes them accessible to the operator GUIs and databases through the EPICS protocol.

The beam current monitoring system shown in Figure 6 consists of the following:

•  The AC Current Transformer (ACCT) measures the beam current at the end of LEBT.

•  A signal condition board (ACCT-E-RM) converts the signal from the ACCT to a signal that can be
handled by a data acquisition device.

•  The MicroTCA controller digitises the signal and propagates it to the EPICS layer.

•  The IOC processes the signal data and makes it available for all other IOCs and services on the
Facility Integration layer. This means, the signal data becomes available for users for monitoring,
analysing, displaying or storing in the control system archive.

The MicroTCA controller is also connected to the fast beam interlock system, which is part of the machine
protection function. Based on the information received from the beam current monitoring system, the
fast beam interlock system can inhibit proton beam production.

The timing system time stamps the readings of the beam current monitoring system. This allows to
correlate them with readings of other systems in the facility. In addition, the timing system enables
facility-synchronous event triggers with deterministic timing. The timing system also broadcasts real-time
information such as accelerator operation modes which is used for on-the-fly configuration adjustments.

15
5.2. Controls Configuration Management
The ESS control systems consist of thousands of local control
systems and their interfaces, connecting to a wide range of
physical equipment. Achieving reliable, repeatable, flexible and
efficient configuration management for all ESS control systems
is a significant challenge in itself.
The ICS division has developed advanced automated processes and tools for managing the configuration and deployment
of EPICS IOCs, PLC code and other devices. The following software components, based loosely on concepts developed in
association with the EPICS community, are involved in the process.

The Configuration Controls Data Base

To manage the very complex configuration of the ESS control systems, operators and engineers work with an abstracted
model of the facility and the control systems. By modelling the control systems, operators and engineers can efficiently
configure the local control systems in detail and then automatically replicate that configuration to be applied on other
identical systems with a few simple commands.
The Controls Configuration Data Base (CCDB) contains configuration information relevant for controlling physical and logical
devices such as cameras, power supplies, pumps, valves, etc. This information is aggregated in the CCDB in one model of the
ESS. The information in this model is complemented by supporting software products such as the ESS Naming Service for
unique identification of devices and the ESS Cable Database which tracks all connectivity in the facility.

ESS EPICS Environment “E3”

An important delivery from ICS is a standardised EPICS development environment that is distributed to stakeholders, in-kind
partners and commercial suppliers.
The development environment - the ESS EPICS Environment, or E3 - is a full software development environment for
developing and deploying EPICS EOCs. E3 contains the correct EPICS base, device support modules, etc. for the ICS-
standardised hardware platforms and operating systems.
With E3, contributing developers around the world can make sure that their software contributions are fully compatible with
the ICS deployment environment.

Configuration Management and Deployment

The E3 code base and facility information from the CCDB is used by a set of tools that assist engineers and operators to
quickly and consistently deploy complex configuration to the ESS control systems.
Engineers and operators first create or edit the system configuration in the CCDB to model the control system. A service
called IOC Factory enhances the engineering efficiency by translating information in the CCDB to deployable configuration.
Here, configurations can be efficiently managed through replicating or sequencing commands.
Operators can extract configuration information from the CCDB and apply adjustments to selected parts of the control
system with a few commands. Proper versioning and auditing of changes in control system configurations are also possible
through the IOC factory tool.
A similar tool specialised for industrial automation systems, called PLC factory, helps engineers reduce time consuming
tasks by automatically generating I/O lists and standardised function blocks from information in CCDB.

Channel Finder
Another important product is the EPICS channel finder application. It implements a directory service for control system data,
which allows users easier browsing and aggregation of entities in the control system. An estimate is that the full production ICS
will contain 1.6 million values that are tracked by ICS. The channel finder helps users navigate and customise groups of interest.

16
5.3. Machine Protection
The Role of Machine Protection at the ESS.

ESS, being a user facility for neutron science, is facing high neutron beam availability requirements and is
largely relying on custom made, specialised, and expensive equipment for its operation. Damage to this
equipment could cause long shutdown periods, induce high financial losses and, as a main point, interfere with
international scientific research programs relying on ESS operation and related beam production. Implementing
a fit-for-purpose machine protection concept is one of the key challenges in order to mitigate these risks.

Achieving High Operational Availability of the ESS

The availability of the ESS facility an important parameter measuring the average neutron production compared
to the planned production for a certain time period. Availability characteristics for a system can be determined
by its reliability, availability, maintainability and inspectability. The ESS equipment relevant for neutron beam
production, in this context called “the Machine”, is in the scope of Machine Protection engineering. The
Safety and Protection Group is taking the leading role in the engineering management of ESS-wide Machine
Protection, and in the design and implementation of dedicated Machine Protection Systems (MPS).

The goals for machine protection are formulated as follows:

Goal 1: Goal 2:
Machine protection shall, in that order, prevent Machine protection shall protect the machine from
and mitigate damage to the machine, be it beam- unnecessary beam-induced activation having a
induced or from any other source, in any operating potential to cause long-term damage to the machine
condition and lifecycle phase, in accordance with or increase maintenance times, in accordance with
beam and facility related availability requirements. beam and facility related availability requirements.

Means to achieve Machine Protection

The availability goals will be achieved through:

•  Designing the systems relevant for neutron beam production with high inherent reliability and overall low damage potential,

•  Minimisation of the mean down time of these systems by introducing dedicated technical systems preventing
and mitigating damage, called Beam Interlock Systems and Machine Protection Systems,

•  Minimisation of the mean down time of the ESS systems by introducing dedicated operational and preventive
maintenance procedures reducing the probability for unscheduled corrective maintenance.

•  Supporting systems dedicated to reducing mean down time. These include analysis, management and recovery
tools addressing operational activities related to machine protection (such as for customised for “post-mortem”
event analysis).

The right strategy to achieve the availability goals will involve a mix of the above measures. The ESS approach
for achieving the operational availability goals using the aforementioned measures is presented in more detail
in the ESS Machine Protection Concept [15].

17
5.4. Main Control Room and
Experiment Operation Consoles
The ESS systems are operated in parallel from various
workplaces on the site. Most prominent is the main control
room, which centralises the human operation of the proton
accelerator and the target station and provides a place for
supervision for the facility as a whole.

Figure 7: ESS Main Control Room in a virtual reality model

The main control room is planned to be manned with an accelerator, target and ICS specialist as permanent
operators and to provide workspaces for up to approximately 25 persons. The main control room layout
includes 13 permanent workspaces with partially fixed-function desks, such as for the operation of the
Personnel Safety Systems. Additional workspaces and a meeting room are available, as shown in the back
of Figure 7. Neutron experiments are not operated from the main control room, but from ICS-provided
consoles in the experimental areas, which are customised to the particular experiment needs.

18
5.5. Personnel Safety Systems

Providing and ensuring safe conditions for ESS personnel is

essential. The Safety and Protection Group in ICS division is
responsible for functional safety realised by electrical, electronic
and programmable electronic safety related systems, which at
ESS are called Personnel Safety Systems (PSS).

The Safety and Protection Group is responsible for developing and operating all of the Personnel Safety
Systems, including design, manufacturing, commissioning and maintenance. The Personnel Safety Systems
protects personnel from the facility’s prompt ionising radiation hazards generated by particle acceleration or
radiofrequency systems. Furthermore, the Personnel Safety Systems identify and mitigate other hazards such
as cryogenic hazards (burns, oxygen deficiency), oxygen depletion hazards, high voltage hazards, magnetic
field hazards, laser hazards and motion control hazards.

“PSS controlled areas” are identified as hazardous areas where personnel might be affected, such as the
accelerator tunnel, target utility rooms, the target bunker and areas at the neutron instruments.

The following types of personnel safety systems are realised:

•  Safety interlock system: This personnel safety system ensures that access to controlled areas is
prohibited if the beam production is permitted. Conversely, if access to controlled areas is allowed,
beam production is not permitted. In the case of an emergency, all hazardous equipment under
control is switched off. Emergencies are triggered by events such as elevated dose measurements
and intrusions to PSS controlled areas.

•  Access control systems monitor the conditions for enabling the entry stations to “PSS controlled
areas” and control door subsystems for openings and access.

•  Oxygen deficiency hazard monitoring systems alarm personnel by activating signalling lights and
sounders.

The Personnel Safety Systems are developed in compliance with the international functional safety standard
IEC 61508 [23], a widely adopted practice within the accelerator based research domain.

19
6. ICS PRINCIPLES
The ICS division uses technologies and engineering methods
from different control system domains in order to achieve the
overall best results for ESS.

To exploit the strengths of different technologies and engineering domains to the best benefit, the ICS division applies
the systematic approaches to system life cycle management. Each approach is tailored to utilise the domain-specific
best practices, which are often described in standards or engineering literature. The ICS approaches are continuously
harmonised with the overall ESS engineering management, particularly as described in the ESS Handbook for
Engineering Management [19].

Most ESS control systems enable conventional functions, including control for beam production and related enabling
systems such as cooling, studies of machinery behaviour, neutron experiment execution, experimental data acquisition,
processing, storage and analysis. The integration of ESS systems realised by such conventional control systems
requires well-organised development, installation and transition to operation. This is executed according to the
system life cycle model described ICS Integration Strategy [17]. This approach is based on the systems and software
engineering standard ISO 15288 and the closely related INCOSE Systems Engineering Handbook, thus implementing
best practices in Systems Engineering for complex systems.

Machine Protection is a major factor for reaching the ESS neutron production availability goals. To address this
mission-critical relevance appropriately, a special approach to life cycle management has been developed and is
applied to the Machine Protection Systems-of-Systems [25]. It combines concepts and principles from the functional
safety domain based on the IEC 61508 [23] and IEC 61511 standards with Systems-of-Systems Engineering. The
approach is explained in detail in the “Machine Protection – Systems Engineering Management Plan” [16], which
also clarifies the relation to local protection systems and constituent systems such as the ion source, proton beam
choppers, various sensor systems etc. which participate in realising Machine Protection at ESS.

The Personnel Safety Systems are safety-critical, and thus the development, operation and maintenance are executed
in compliance with suitable international standards for functional safety engineering. The Safety and Protection Group
in the ICS division applies the IEC 61508 [23] and IEC 61511 [24] standards for the development of the ESS Personnel
Safety Systems. Using these standards is a widely adopted approach for safety-related systems in the accelerator
research facility community. The approach is described in the PSS Development and Quality Assurance Plan [20] and
in the ESS Rules for Configuration Management for Personnel Safety Systems [21].

The Control Software Group organises the development of software according to the character and complexity of
the developed software system. For complex software services, the development life cycle approach is following the
traditional V-model, with explicit development-stage related documentation. In other cases, such as user application
development or feature additions, an agile approach is applied. This differentiation allows the Control Software Group
to adapt towards expectations on response time to feature requests, stakeholder inclusion in the development
process, documentation and architecture quality and efficiency in resource usage.

In Agile developments, the Control Software Group creates a first working software version with limited functionality
as early as possible. This prototype is then continuously evaluated with the stakeholders, for example by ESS physics
and engineering groups. Based on this feedback, incremental improvements are implemented in the software. The
Agile loop is repeated until the software realises the desired functionality. These iterations are managed in daily to bi-
weekly stakeholder meetings. Software changes during operation due to new feature needs, technical adaptations or
bug fixes are realised by further Agile loop iterations.

20
7. ICS ORGANISATION
The matrix organisation applied in ICS division relates the ICS
line management to the ICS project management.

At the heart of any efficient organisation is the L i n e manag ement fun c ti on that builds permanent groups and
temporary teams, secures commitment and motivation in those teams to provide the necessary competence
and capacity needed for the organisation to deliver. To execute activities in the organisation in a structured and
systematic way, line management creates a Proj ec t man ag ement fu nc ti on and hands the executive mandate
to the project managers.

d Ho w
W ho an
Line = When
h at and

7.1. Line organisation

t = W
Projec

The ICS division is organised hierarchically in a group structure.

Four domain competence groups exist:

The Hardware and Integration Group The Control System Infrastructure Group
The Hardware and Integration Group develops, The Control System Infrastructure Group designs,
implements, operates and maintains the non-safety- implements and maintains the backbone data
critical control systems of the ESS, including the and communication networks for the control
EPICS integration and timing system. The HWI teams and timing systems, the ESS control rooms,
also engages in customising base technologies, data centres and other equipment required for
such as industrial automation and MicroTCA based the reliable operation of the control systems.
systems, for its application by stakeholders. The group provides operations workstations,
servers and file storage required for operation
of the facility. This group is responsible for
The Safety and Protection Group implementing the ESS main control room, which
allows the facility to be controlled from a central
The Safety and Protection Group coordinates and location.
executes the development of highly dependable
systems. For enabling a high neutron beam
availability, the group coordinates the realisation of
machine protection functions at ESS and executes The ICS Management Team
the development, operation and maintenance of The ICS Management Team executes and supports
dedicated machine protection systems. For worker line and project management functions, and
safety, the group develops, operates and maintains establishes consistent architectures, technology
the ESS Personnel Safety Systems. standards and systems engineering for the ICS
division.

The Control System Software Group Each group is viewed as a separate organisational
unit with its own line manager and organisation
The Control System Software Group provides context description. ICS line managers are
software toolsets needed for the management and responsible for developing and maintaining the
operation of technical ESS systems. The group’s strategy for keeping the ICS division organisation
scope is software such as controls configuration concurrent with ESS needs. This strategy is
management tools, alarm handling tools, systems formulated in the ICS organisation context
for troubleshooting and documentation of the description[1]. Figure 8 shows the organisational
machine history and physics-related applications. chart of the ICS division.

21
 Henrik Carling
Division Head

Karl Vestin Susanne Regnell Annika Nordt Remy Mudingay Anna Gillberg
Hardware and Integration Controls Software Safety and Protection Controls Infrastructure Team Assistant

Hector Novella
Deputy Project Manager

Angel Monera Ben Folsom Alberto Toral Diez Benjamin Bertrand Thilo Friedrich
FPGA Engineer PhD Student Technician Software Engineer Systems Engineer

Benedetto Gallese Banafsheh Hajinasab David Sánchez-Valdepeñas Johan Christensson Timo Korhonen
Integrator Software Engineer Automation Engineer Infrastructure Technology Engineer Chief Engineer

Faye Chicken Claudio Rosati Denis Paulic Susann Skarin Maria Romedahl
Technician Software Engineer DGL, PLC Engineer Junior Network Technician Technical Coordinator

François Bellorini Dirk Nordt Enric Bargalló Anders Harrisson Wojtek Fabianowski
Integrator Software Engineer Lead Analyst Software Configuration Manager
In-Kind Manager

Javier Cerejo Emanuele Laface Fernando Carrasco Martin Fredrik Luthander

PhD Student Accelerator Physicist Technician Software Engineer

Jeong Han Lee Fredrik Söderberg Manuel Zaera-Sanz Peter Holgersson

Integrator Software Engineer Senior Machine Protection Engineer Electrical Engineer

Joao Martins Georg Weiss Mattias Eriksson

Integrator Software Engineer Technician

John Sparger Jan-Åke Persson Morteza Mansouri

Integrator Senior Software Engineer Lead Integrater PSS

Julen Etxeberria Juan Esteban Müller Riccard Andersson

Junior Controls Engineer Software Scientist Technical Project Coordinator

Peter van Velze Karin Rathsman Stephane Gabourin

Technician Accelerator Scientist Lead Engineer Fast Interlocks

Saeed Haghtalab Ricardo Fernandes Stuart Birch

Integrator Control System Software Architect Senior PSS Engineer

Simone Farina Lars Johansson Szandra Kövecses

Embedded Systems Engineer Software Engineer Lead Integrator Machine Protection

Thomas Fay Viktor Fred

Integrator Lead Engineer Electrical Installations

Tomasz Brys Yong Kian Sin

Integrator IEC61508 Controls Engineer

Johannes Kazantzidis Paulina Skog

Integrator Document Specialist

Marino Vojneski Meike Rönn

Integrator Document Specialist

Mehdi Mohammednezhad
Integrator

Michael Beck
Work Package Manager

Miklos Boros
Integrator

Nicklas Holmberg
Integrator

Nour Akel
ICS Installation Coordinator

Oliver Talevski
Embedded Engineer

Philippe Rabis
Work Package Manager

Wayne Lewis Employees 51

EPICS Integrator

Consultant 18

Figure 8: ICS Organisation in Q2 2018

22
7.2. ICS Project Management
The ICS Project managers derive executable, scheduled tasks
with well-defined budgets based on stakeholder, technology
and standardisation needs.

Project managers also ensure that ICS tasks are aligned and coordinated with activities of the other ESS
divisions. The ICS construction project scope is organised and managed in work packages (see 7.3). These are
arranged by

•  the integration of ESS systems and equipment for the ESS operation and maintenance (work packages
05, 07, 09, 10, 11, 12, 13),

•  the provision of software based applications for operation, maintenance and configuration management of the
ESS (work packages 03, 08),

•  the provision of customised technologies for the ESS (work package 04),

•  general ICS management (work package 01 and 06),

•  test and prototyping activities (work package 14).

The ICS long-term planning and budget allocation is managed in the central ESS planning tool Primavera P6,
where the ICS plans are linked with plans of other ESS divisions. Detailed tasks and daily work is managed using
the JIRA on-line collaboration tool, which supports the agile methods used in software development and also
enables issue tracking, which is often used during installation and commission.

23
7.3. Construction Project Organisation
The tasks of ICS division for the ESS construction project, as
stated in the ICS project charter [6] is organised in work packages:

Table 1: ICS Work Package Overview

01 Management
Central management services for the ICS division including in-kind management

03 Software Core
Development and integration of software core components for the integrated control system

04 Hardware Core
Selection, preparation and delivery of standardised electronics hardware equipment for control, data acquisition and timing

05 Machine Protection System

Design, install and commission the ESS machine protection systems

06 Equipment
Administrative work package to collect all equipment management in ICS. No activities

07 Control System Infrastructure

Control rooms and the control system infrastructure including data networks, computing facilities

08 Physics
Development of support tools for machine commissioning and operations based on a mathematical model

09 Personnel Safety System

Design, install and commission all personnel safety systems for accelerator, target, and neutron instruments

10 Integration - Accelerator
Accelerator controls development and integration

11 Integration - Target
Target station controls development and integration

12 Integration - Instruments
Neutron instrument controls development and integration

13 Integration - Facilities
Development and integration of controls for building management systems such as cooling water and air conditioning

14 Test Stands
Design, construction and maintenance of relevant ICS test stands and laboratory facilities

24
7.4. ICS Information Management
The ICS division uses ESS tools for Information Management.

The ICS Division uses the applicable ESS Information Management systems to enable access and manage
human-processed engineering information in the corresponding contexts:

•  ICS Project Information repository. [8]

•  ICS Line Management repository. [9]

•  Technical specifications of ESS/ICS systems: ESS Breakdown Structures [11]

•  Asset management: Infor EAM. [12]

•  Collaboration tool: Confluence and JIRA. [7]

•  ESS schedule and budget planning: Primavera P6

•  Engineering processes and conventions: ESS Management System. [10]

7.5. Financial key figures

Budget, in-kind contribution and risks.

The total ICS construction project budget allocated for implementing the full ICS construction scope is in the
order of 80 M€. A certain proportion of this budget should be utilised as in-kind contribution from ESS member
states. At the beginning of 2018, the ICS project had achieved a level of about 20% in-kind contribution. A re-
baselining of the ESS construction project plans was ongoing in the first half of 2018, which may imply changes
also to the ICS context.

ICS is applying proper project management practices, including risk management. The ICS risk management
plan [26] is used for keeping track of a risk register and treatment actions.

25
8. ICS VISION, MISSION AND VALUES

ICS Vision
ESS operated efficiently, reliably and safely, with a control system that everyone loves.

Mission

To provide and maintain world-class and cost-efficient control, protection and safety systems and services for
the ESS facility. To develop competence and innovative solutions that can be shared in the community through
open processes.

Behaviours and Values

ICS staff and consultants follow the ESS core values [2] and the ESS code of conduct [4]. The ICS leadership
and management team also implements the following behaviours as described in the ICS Organisation Context
Description [1].

•  Lead by example, Generate energy, Communicate

•  Simplify, Push for excellence, Get results

•  See the big picture, Challenge, Drive one ESS

ICS Simplified Values

The ICS division uses the following simplified values as a behaviour guide in the daily work

•  Be nice!

•  Use common sense!

•  Have fun!

26
9. REFERENCES
[1] ICS Organization Context Description. ESS-0053845

[2] ESS Core Values. [TBD]

[3] ESS Managers Handbook. [TBD]

[4] ESS Code of Conduct. ESS-0002208

[5] ICS Organisation Chart. ESS-0081625

[6] ICS construction project charter. ESS-0066381

[7] ICS Confluence Space. https://confluence.esss.lu.se/display/ID

[8] ICS Project Repository. CHESS link

[9] ICS Line Repository. CHESS link

[10] ESS Management System. CHESS link

[11] ESS Facility Breakdown Structure. CHESS link

[12] ESS Asset Management tool. [TBD]

[13] ICS Handbook. ESS-0067637

[14] ESS Timing System. FBS link

[15] ESS Machine Protection Concept. ESS-0035197

[16] Machine Protection Systems Engineering Management Plan. ESS-0057245

[17] ICS Integration Strategy. ESS-0054678

[18] ESS Naming Convention. ESS-0000757

[19] ESS Handbook for Engineering Management. ESS-0092276

[20] PSS Development and Quality Assurance Plan. ESS-0061709

[21] ESS Rules for Configuration Management of Personnel Safety Systems. ESS-0058389

[22] EPICS. https://epics.anl.gov/

[23] IEC 61508-2010. Functional safety of electrical/electronic/programmable electronic safety-related systems.

[24] IEC 61511-2016. Functional safety – Safety instrumented systems for the process industry sector.

[25] Systems of systems engineering for particle accelerator based research facilities: A case study on engineering
machine protection. Systems Conference (SysCon), 2017 Annual IEEE International. DOI: 10.1109/
SYSCON.2017.7934806

[26] ICS Project Risk Management. ESS-0052287

27
10. DOCUMENT REVISION HISTORY
Revision Reason for and description of change Author Date

1 First issue T. Friedrich 2018-04-27

28