Interoperability of System Protection Software

Jared Gurley Luci Hays David Ridley Zachary Austin

Oncor Electric Delivery Oncor Electric Delivery Oncor Electric Delivery Oncor Electric Delivery
Fort Worth, TX USA Fort Worth, TX USA Fort Worth, TX USA Fort Worth, TX USA
[email protected] [email protected] [email protected] [email protected]

Nathan Thomas Luke Hankins Alberto Borgnino Joe Perez

SynchroGrid SynchroGrid SynchroGrid SynchroGrid
College Station, TX USA College Station, TX USA College Station, TX USA College Station, TX USA
[email protected] [email protected] [email protected] [email protected]

Abstract—Software is used in system protection for a variety I. I NTRODUCTION

of reasons such as data management, performing short circuit
simulations, and automating tedious work, all aiming to empower There is growing consideration in many system protection
engineers to continue developing reliable protection schemes teams to embrace software automation as a foundational
for an increasingly complex power grid. We have found that component of relay settings processes. While basic, ad hoc
interoperability between software packages such as the short
circuit model, relay setting database, and automated relay setting automation (e.g., scripts for data gathering and manipulation)
calculation software, enables sophisticated, streamlined processes. have for some time been used to augment existing processes,
With seamless interaction between software, data is accurately some utilities are choosing to take a more holistic view of how
and efficiently exchanged, engineering resources are less strained, ongoing software advancements in system protection might
and compliance requirements such as NERC PRC-027-1 [5] R1 inform a change in process.
and R2 are more easily met.
We present a case study of Oncor Electric Delivery’s software One emerging trend in system protection software that
integration efforts, describing both methodology and benefits serves as a catalyst for process innovation is software inter-
realized. Oncor utilizes multiple software tools which are com- operability. For some time, many industry applications pos-
posed together to implement standardized, automation assisted sessed some built in automation capabilities, such as scripting
workflows for system protection activities. Beyond the standard language in short circuit models. These are used by some
capabilities of each application, the company realized substantial
benefits from these efforts, avoiding the common inefficiencies engineers, but their specialized and proprietary nature have
and errors that exist at the boundaries and interactions of such limited their use in creating maintainable, enterprise grade
tools. This more holistic, integrative approach is made possible by solutions. Attempts at integrated solutions with other software
next-generation application programming interfaces (APIs) that (i.e., relay databases and relay settings development packages)
expose greater access to the capabilities of each software solution. required tenuously cobbling them together with custom data
In addition to more efficient data transfer, these APIs allow
one program to more directly interact with another, requesting management scripts that inevitably break with software up-
actions be performed (e.g., fault simulations) on its behalf. Novel grades and labor turnover.
functionality can thus be created, leveraging each application’s Increasingly, system protection software vendors are ad-
strengths to create a single, automated process. dressing these deficiencies by providing modern application
We present a relay settings development process (PRC-027-1 program interfaces (APIs) which allow their execution to be
R1) with automated settings calculation software that interacts
with the short circuit model, generates reports as well as relay guided and data retrieved by 3rd party applications. This addi-
configuration files, and stores results in the database. Second, grid tion dramatically accelerates innovation, as these packages can
model consistency is ensured by a novel algorithmic approach be easily composed with others, creating integrated solutions
to relay database and short circuit model linking, enabled by built with modern programming languages and libraries. We
integrative analysis of both assets from the database and grid will discuss several integration examples in this paper and
model topology inspection. We next discuss lessons learned and
then describe ongoing work using interoperablity for wide area discuss the new types of solutions they are enabling.
coordination studies (PRC-027-1 R2). This work is based on Oncor Electric Delivery recently completed a software inte-
automated analysis that empowers engineers to visually resolve gration effort which was part of a broader initiative to increase
issues which are then documented in automatically generated, per the efficiency of the relay settings development process. In
relay reports that are stored in the database. We then conclude
addition to existing short circuit software, the company de-
with future work discussion. In summary, we aim to convey that
software integration in system protection is both necessary and ployed relay database software in 2018 followed by automated
beneficial to increase efficiency, reduce errors, and equip utilities relay settings calculation software in 2021. These third party
to handle the increasing complexity of engineering activities. solutions teamed with internal tools are now combined into an
Index Terms—System Protection, Microprocessor Relays, Re- end-to-end, automation assisted process.
lay Settings, Software Interoperability, Process Improvement
Together with interoperability, the decision to move forward
came in the context of a growing familiarity with these soft- to envision what the new process might look like. Next, new
ware packages, settings work growth increasingly outpacing process requirements were developed including the following:
that of personnel, and a revisit of the settings process driven • Uniform - used for all relay settings (transmission and
by PRC-027-1. The goals of the project were: distribution).
• Error Reduction - By augmenting engineering expertise • Familiar - based around how most relay setting engineers
with automation throughout the process workflow com- create relay settings.
mon sources of errors, such as copy-paste, are simply re- • Integrated - utilize the existing relay settings database
moved. Furthermore, by enforcing a standard philosophy and controlled by its statuses and security settings.
across the system, errors caused by variable application • Documented - used as a reference document by all relay
are eradicated. Finally, tedious tasks such as testing points setting engineers.
generation are completely automated. • Measured - extract process data out of the relay settings
• Increased Efficiency - Assistive automation and error database to gain a better understanding of the following:
removal reduces the time required to develop, review, and – Total number times each step was completed.
deploy settings. – Amount of days spent on each process section.
• Reduce Engineer Turnover - By streamlining repetitive – Engineer who is completing each task.
tasks such as fault simulations and report generation, a – Types of errors occurring and counts of each type.
more sustainable process has been established. – Number of times a rework of setting is required.
• Streamlined Compliance - With growing grid complex-
ity and the need to perform frequent, recurring analysis
as part of compliance activities, creating automation
solutions to streamline these efforts is necessary.
• Enable Data Analytics - Whatever is measured can be
improved. An integrated workflow that tracks effort at
each step sets the stage for continuous process refinement. Fig. 1. Original Process Workflow Statuses
The outline of this paper is as follows: In Section II,
we outline the new process for relay settings development B. Analysis of the Existing Process
(PRC-027 Requirement 1) that was developed and then how
The existing process (depicted in Figure 1) called for
interoperability plays a role in its realization in Section III.
engineers to first create a new project in Draft in the relay
We next discuss in Section IV how improvements to the grid
settings database (RSD). The engineer would then progress
model were enabled by software integration efforts and then
the status of the project to Review for a final review, and
discuss lessons learned in Section V. We then briefly discuss
Issued to Field, with each of these transitions happening when
work we are doing for wide area coordination in Section VI
a phase was complete. This linear progression was not in
and then conclude with a discussion of future work.
practice, however, how the group integrated the process into
II. R ELAY S ETTINGS D EVELOPMENT P ROCESS their workflow. Instead, engineers would start a project and
PRC-027 R EQUIREMENT 1 send deliverables to an experienced engineer for review before
ever starting a new project in the RSD. Once the review
One of the most fundamental processes in system protection had been completed and all deliverables had been developed
is new relay settings development. The importance of a settings outside the purview of the RSD, the engineer would start a new
development process that is both documented and followed is a project in draft, upload all their documentation, and mark it for
fundamental aspect of the recent NERC PRC-027 [5] standard. review. Because the review was completed independent of the
In this section, we provide an overview of Oncor’s new process RSD workflow, the Draft and Review statuses in the database
and describe its development from requirements gathering to were logged with times not correlating to the actual time spent
implementation. While we give some mention to the software on the project. The resulting data metrics’ inaccuracy about
tools used here, they and their interoperability are discussed the required effort from the setting engineer and reviewing
in detail in Section III. We will focus on the relay settings engineer made analysis for process improvement difficult and
database (RSD), short circuit model (SCM), and relay settings led to the the Measured requirement of the new process.
calculation (RSC) software interactions. Once the project had been completed and reviewed, configu-
ration files were distributed to the field technicians. The RSD
A. Process Requirements contained a method for the hand off of documentation and
Oncor began developing the new relay settings process with configuration files from engineers to protection and control
a review of the requirements of PRC-027 Requirement 1, technicians. Technicians would receive an email when new
to ensure that the new process would be compliant to R1.1 work was released. The technicians would then download
(review and update of short-circuit model data) and R1.2 the documentation and configuration files from the RSD to
(review of developed settings). This was follow by a review of a companion testing software. This synergy provided a means
the current relay setting process and brainstorming meetings of linking and documenting the test results, any applicable

2
compliance related information, the originally issued settings, the final solution was not infeasible to implement in software
and as left settings, that may include more device specific nor overly burdensome to engineers and other users.
information in the configuration file not available to the
engineer at the time of coordination. While there were clearly
problems in this process with regards to logging the time it
takes to complete engineering projects, the database provided
the relay settings team greater visibility into what projects
Fig. 2. New Process Workflow Statuses
were performed by each engineer, what the status of the relay
settings were for those projects, and provided validation of the
configuration files applied to the relays in the field. Before the TABLE I
adoption of the RSD, receiving as left configuration files was P ROCESS S TATUS C OMPARISON
far less common.
Status Number Old Process Status New Process Status
1 Draft Draft
C. Design of the New Process 2 Review Peer Review
3 Approved Finalize Documentation
Oncor’s definition of new process requirements brought 4 Issued to Field Final Review
clarity as to what the process needed to look like. Designing 5 Approved
it began with drafting in a diagramming application which 6 Issued to Field
allowed rapid, iterative refinement as development progressed
and eventually yielded a more detailed flowchart of the pro- Controls are assigned between two different engineering
cess. The detailed form was necessary to decompose higher security groups, relay setter and relay setter reviewer, and
level steps into either decision steps or side steps. Next, these groups dictate what steps a user is able to approve.
a process document was developed that matched the steps All engineers are able to create a Draft project and move the
of the flowchart but provided in depth descriptions of each status of the project to Peer Review, Finalize Documentation,
step, references, and definitions. After both the flowchart and and Final Review. Approve is limited to relay setter reviewers
process document were created, they were analyzed against who are experienced and perform the final review before the
every type of settings Oncor calculates to ensure that all original engineer issues the documentation and configuration
needed items were included. files to the technicians. The relay settings database captures
Oncor also developed an emergency process which circum- which user progressed the project to each subsequent status.
vents the standard process when speed is of the essence. For As an example, consider three engineers:
familiarity, the emergency process needed to be similar to the • Sarah - Lead engineer on a project at Station A.
normal process yet provide engineers the required flexibility • Phillip - Peer review engineer.
for emergency situations. A primary goal of this process was • Bennet - Region lead engineer and PE for final review.
to reduce the time spent early in the process so that settings
When Sarah begins her project at Station A, she creates a
can be released to field technicians quickly. We observed that
new Draft in the relay settings database. Once her coordination
the review steps required the most amount of time to complete,
and documentation are complete, she moves the status to Peer
apart from the actual relay setting calculations. Hence, in order
Review and Phillip is notified that her work is ready for review.
to both meet the review requirements of PRC-027 R1.2 and
Phillip reviews her work and if he agrees, will progress the
reduce time to release to field, the review step was shifted
project to Finalize Documentation. If Phillip has corrections
to after the field technicians had completed work. While this
for Sarah, he provides her with notes and reverts the status
achieves the desired goal, it does result in more manual steps
to Draft, where the process restarts. Once the project is in Fi-
and higher total time to complete the process.
nalize Documentation, Sarah will complete her documentation
and configuration files and progress the status to Final Review
D. Scoping and Analysis of Process Controls
which is performed by Bennet. If Bennet agrees with the
Because the relay settings engineering group was already documentation and coordination, he will progress the status to
using the relay settings database, the clear path forward was Approved. If Bennet has documentation corrections for Sarah,
to implement process controls there, leveraging its rigid status he will revert the status to Finalize Documentation and if he
controls and the persistent, automatic storage in the database has any major coordination corrections he will revert the status
structure. This approach provided many gains over other to Draft where Sarah will start the process again. Figure II
methods considered, including minimization of training due shows the name captured by the relay settings database as the
to existing workflow and software familiarity. project moves through its lifecycle.
1) Process Controls Development: During the new process The workflow provides a log of each engineer’s work. As
development, the limitations and existing structure of the relay all data is in the database, metrics are generated to gauge the
settings database were evaluated. Pragmatic consideration of development of engineers, as their revision counts drop over
how engineers would interact with the process also greatly in- time. By logging the user, time, and date of the progression, we
fluenced the design of process controls. These efforts ensured can perform further analysis to quantify the average amount

3
 TABLE II and perform engineering work. An additional process require-
S TATUS NAME L OGGING IN THE DATABASE ment, measurable, is also enabled by this work, with the end
Status Name DB Log goal for all project work from initial project creation to final
Draft Sarah issuing to be saved and tracked in the relay settings database.
Peer Review Sarah Next in Section III, we will describe how software interoper-
Finalize Documentation Phillip
Final Review Sarah ability plays a significant role to meet these two requirements,
Approved Bennet as well as others such as uniform and documented.
Issued to Field Sarah
III. S OFTWARE T OOLS AND I NTEROPERABILITY FOR
PRC-027 R EQUIREMENT 1 P ROCESS
of time spent per project type per step per engineer. This We now describe the software tools used to implement On-
provides greater insight into the distribution of projects across cor’s new process, specifically focusing on the ways in which
the engineering group and across the year. These insights they interact with each other for end-to-end process efficiency.
enable the optimization of work distribution across engineers. There are three external tools Oncor employs together with an
With the addition of Peer Review, we increase reviewer internal spreadsheet tool that are composed into a cohesive,
diversity. Previously, only experienced engineers reviewed integrated solution for process automation. Below are the three
projects. Now, new engineers review documentation and co- vendor tools Oncor chose for this process:
ordination from more experienced engineers, gaining experi- • Short Circuit Model (SCM) - ASPEN OneLiner V15 [6]
ence, preventing isolation amongst the engineering group, and • Relay Settings Database (RSD) - PowerBase V7 [11]
promoting the sharing of best practices. • Relay Settings Calculator (RSC) - SARA V3 [13]
2) Implementing and Testing New Process Controls: Tran- While these offerings were chosen both because they met
sitioning to the new process began in a pre-production environ- functional requirements and the vendors’ ongoing collabora-
ment for testing. This pre-production environment was a copy tions to promote interoperability, the principles discussed in
of the production environment to ensure the process changes this paper are general and should be achievable with other
would work with the data it would eventually be deployed system protection software exhibiting similar features and
on. We began by converting the names of statuses as detailed modern application programming interfaces.
above. Because of the limitations of the way our previous In Figure 3, we present a simplified overview of the software
process was created, some historical status values needed to workflow both the external and internal software tools employ
be migrated. For example, Status 3 was previously Approved to implement Oncor’s new settings process. Briefly, the process
and was now Finalize Documentation, and therefore the status begins with a Draft project created in the database, from
number was moved to 5 to maintain consistency (see Table I). which existing settings have been placed in the model as
After status migrations, security group permissions were described in Section IV. Next, the settings calculator retrieves
applied and validated. For the emergency process, a check a specification of the protection philosophy in the form of a
box was implemented on each item in the database, and an template from the database and interacts with the short circuit
additional status named Emergency Issue was added to allow model to create settings and populate the spreadsheet tool. The
it to initially skip the review steps. The check box also serves spreadsheet is then used together with the settings calculator’s
as an indication on a relay setting that needs to be followed review module to perform the settings review. All artifacts
to ensure that the emergency process gets completed. Testing created during the development process are then stored in
took several months and ensured that the process flowed the settings database and issued to the field. We describe
logically as well as achieved the requirements set out from software interactions in detail in the following subsections and
our analysis of PRC-027. then describe further process refinements uncovered during the
E. Deployment of the New Process integration project that leveraged the synergy made possible
by the software packages’ composition.
Once the new process testing was complete, a process
change date was chosen and engineers were advised to A. Short Circuit Model and Relay Settings Calculator Inter-
complete projects using the current process, to avoid delays operability
providing settings to field technicians. During the production
outage, the new process was placed into production in a similar The fundamental functionalities of a relay settings calculator
manner to that of the pre-production environment. Engineers used to implement the process described in this paper are:
were trained on the new process the next day. The final result • Formally define a utility’s protection philosophy and

was a well-documented, generalized normal process that met compliance checks in a customizable manner.
the requirements listed above as well as an emergency process • Automatically interact with the short circuit model to

that provides the flexibility to resolve emergency issues. gather information such as:
To satisfy the integrated process requirement, the workflow – Grid Topology - Source lines, remote lines, multi-
was designed to limit how often engineers would jump be- terminal configuration, tap buses and lines, etc.
tween applications and locations to find standard information – Grid Characteristics - Impedances, line ratings, etc.

4
 Fig. 3. Simplified Software Process Workflow

– Fault Derived Calculations - currents, impedances, scalable programming languages for systems design. Using
relay operation points, contingency analysis, etc. this environment allows us to interact with both programs
• Provide an interface to evaluate calculated settings, adjust seamlessly and leverage tested and maintained libraries such
as necessary, and document justifications. as the Boost Graph Library [8] we use to represent power grid.
• Automate ancillary activities such as testing points to Complex, multi-bus substation analysis, generalized multi-
further reduce engineering effort. terminal application support, and contingency analysis such as
• Streamline reporting and compliance documentation. strongest source detection are all implemented using generic
• Generate relay files to avoid copy and paste style errors. graph algorithms [9], drawing from the fundamentals of com-
puter science. Were it not for software interoperability, such
We have previously described our approach to implementing sophisticated solutions would not be feasible. Maintainability
the settings calculator (RSC) and its interactions with the is assured by using documented, stable APIs between the
short circuit model in [1]. Figure 4 summarizes the packages’ programs that allow them to safely interact with each other,
interactions. In this section, we describe recent advancements creating novel functionality to automate settings development.
made possible by a newer API to the short circuit model
One example of new functionality built during this project is
(SCM).
testing points creation using a customizable template. The RSC
dynamically calculates the operation point (i.e., reach) of all
distance and overcurrent elements and the corresponding char-
acteristic impedances are used in symbolic mathematical ex-
pressions to create a testing point for a given element. For ex-
ample one might choose to specify the testing point for a Zone
2 distance element to be (OpZ (Zone1) + OpZ (Zone2))/2,
representing a point halfway between Zone 1 and Zone 2
reaches. Using this specification, testing points will then be
generated for every line relay application. This expressive
capability can be adjusted as the process changes in the future,
and in Section III-D2 we describe how the settings calculator
presents the information to the settings engineer for review.

B. Relay Settings Database and Relay Settings Calculator

Interoperability
Fig. 4. Relay Settings Calculator and Short Circuit Model Integration. In addition to the foundational capability of storing relay
settings and other grid asset data, some relay databases provide
The latest iteration of the interface between the RSC and the capability to specify workflows, which as discussed earlier,
SCM is written in modern C++20 [7], an ISO standard allow Oncor to define the steps of the process and trigger
which is one of the most prevalent, high performance, and automated activities as work progresses. In these integra-

5
tion efforts, we leveraged interoperability between the relay The second worksheet Questions presents questions with
database to remove the manual transfer of settings related data. dropdowns for answers, acting as a logic calculator that
For the settings development process, there are two points of compares answers against a lookup table to make appropriate
interaction. First, philosophy templates are retrieved from the setting changes, such as advanced reclosing logic equations or
relay settings row in the database and used as the protection circuit breaker classification. Though the questions are quite
standard for settings calculation. Second, when calculations comprehensive, some settings require further analysis. A Verify
are complete, all digital artifacts of the process are efficiently worksheet is presented next that contains a list of settings that
uploaded to the database and can trigger subsequent actions the setter will manually verify with a single click.
such as the review step of the process. These artifacts include: The last two worksheets in the workflow are the Front
• Settings calculation sheets (discussed in next section) Page and My Settings. They are an accumulation of the work
populated with settings data. performed on previous sheets which is presented in an easy to
• Relay configuration files (rdbs) with developed settings. read format for documentation and will be referenced to create
• Relay Settings Calculator software native files both a relay download file and setting sheet for the settings. In
While this is a relatively straighforward use of the database’s the past, creating these documents represented a large portion
new API, this integration reduces another potential source of of the settings engineer’s time on a project.
errors and increases efficiency by removing several manual The workbook development focused heavily on maintain-
steps previously required by the settings engineer. There are ability for future use within Oncor, with the goal of making it
additional interoperability gains between the relay database simple for engineers revising the standard to make additions.
and settings calculator software for model maintenance. We Furthermore, by integrating manual settings development and
discuss those in Section IV-B. automated approaches into a unified workflow, the broader
adoption of new automation technology can occur without
C. Relay Settings Database and Short Circuit Model unnecessary modifications to the process.
In order to ensure that relays are properly modeled during 2) Interoperability with Settings Development Software:
fault simulations in the short circuit model, it is imperative While the Excel tool provides sufficient flexibility to create a
that the latest settings from the relay database are loaded settings development workflow that is familiar and uniform,
there. Discussion of this critical interoperability component is automating the calculation of core protection settings requires
discussed in Section IV. There we find a sophisticated multi- a level algorithmic expertise and interaction with the short
package integration involving all three vendor solutions that circuit model that is only achievable in the relay settings
proves effective not only in creating a maintainable model, but calculator. These capabilities include complex power system
also discovering some model discrepancies. typology analysis, a flexible fault simulation engine, and
precise tracking of deviations from the standard philosophy.
D. MS Excel Settings Tool and Interoperability with Relay The Excel tool and the settings calculator are interoperable,
Settings Calculation Software as the RSC can both read from and write to Excel work-
During process development, Oncor determined it was an sheets. After automating short circuit model interactions to
ideal time to change from a document standards (MS Word) generate settings, the RSC injects worksheets into the standard
to spreadsheet standards (MS Excel), both to save time and spreadsheet which include settings, underlying calculations,
reduce the potential for errors. These two benefits come from engineer notes, and a summary of settings changed from the
automatic calculation, interoperability, and the flexibility of standard with justification text. Additional information such as
a built in scripting language VBA. In this section, we de- mho graphs and dynamically computed testing points (both in
scribe how interoperability between the Excel workbook tool human readable and testing set formats) are also generated
and the relay settings calculator software meets the process by the RSC and placed in the Excel tool. These actions,
requirement of familiar while meeting overall goals of error coupled with seamless uploading to the relay database, remove
reduction, increased efficiency, and streamlined compliance. a significant source of copy/paste errors and labor by the
We first provide an overview of the tool and then explain settings engineer.
interoperability with the RSC. Once draft settings are prepared, reviewing activities are
1) Excel Tool Overview: The Oncor excel workbook stan- streamlined by a custom review module in the settings calcula-
dards consist of worksheets that each perform a different step tor developed during this integration effort. The review module
of the transmission line relay setting process. On the initial provides a framework for the review engineer to manually ver-
worksheet Fill In, the settings engineer inputs the majority ify critical calculations, providing an essential layer of check
of data that will then be propagated where needed through to the automation assisted settings development process. The
the standardized workflow. This data is both project specific, module’s review process is guided by a customizable review
manually entered data as well as calculated values that relay template which provides the flexible for Oncor to maintain
setting calculator can inject into the worksheet. For engineers it in the presence of future process changes. Importantly, the
or contractors not using an automated calculator yet, these module can read as input both the RSC’s native file format as
calculations can be performed and entered manually, helping well as the Oncor spreadsheet standard. Finally, it can both
achieve the uniform process requirement. read input from and write the results of the review directly to

6
the relay database, further streamlining the overall process. 3) Data Analytics: After building the relay setting process
The interaction of the utility’s internally created Excel into the relay setting database and running our first report
settings tool and a 3rd party relay settings calculation solution on the data, it became evident that many insights could be
demonstrates the importance of interoperability in the adoption mined about the setting group’s work using data analytics. A
of new technologies. The seamless transfer of data between large point of interest was quantifying the amount of work
them both allows familiar tools to leverage the sophisticated being performed and by whom. We were also able to gain
analysis and tight integration the RSC shares with the relay insights around how work is distributed throughout the year.
settings database and short circuit model. This transfer ensures Finally, we found that engineers could use the comment field
that errors are not introduced into the process and creates on database settings rows to track errors or other issues found,
a synergy in their composition that results in significant allowing us to gain understanding around what causes errors
automation gains to the overall settings development process. to be made and to better understand the issues engineers face.
We discuss some of the findings from these analytics further
E. Additional Benefits in Section V-A and expect future analysis of such data to help
drive further process efficiency gains.
Further synergies were quickly discovered during the pro-
cess development and software interoperability design phases IV. M ODEL I MPROVEMENTS
of the project. These benefits include streamlined NERC PRC- As Oncor territory grows and more stations are built, it is
023 compliance, improvements in providing relay loadability increasingly important to maintain an accurate and reliable
data to our facility rating teams, and better data analytics. We system model. The model currently consists of over ten thou-
next describe each of these benefits briefly below and expect sand relay elements at approximately two thousand locations.
further such ideas to germinate with further experience using Maintaining relay settings accuracy with the continual growth
the new, integrated process. and change is simply infeasible to attempt by manual means.
1) Streamlined PRC-023: During API development be- The chosen short circuit model and relay database have an
tween the relay settings calculation software and the relay existing interface to allow the relay linking, allowing settings
settings database, we innovated an improvement to the NERC stored in the database to be seamlessly transferred to relay
PCR-023 compliance documentation process. This stemmed elements in the short circuit model. Once such links are
from the RSC’s ability to calculate and directly transfer data created, a bulk import command allows a rapid refresh of all
that is pertinent to comply with PRC-023 into customizable relay settings in the model. During this project, we created
locations (e.g., asset nameplate fields) in the database. A relay linkings for all transmission relays in the model. The
data location was thus built in the relay setting database that relay linking process requires two activities:
is automatically populated during every setting development • Settings Translation Scripting - The short circuit model
project, in a manner specified in the setting calculator soft- provides a scripting language to define how externally
ware’s template for Oncor’s protection philosophy. stored relay settings can be converted into a relay in the
With this small addition, the latest PRC-023 data will be short circuit model.
populated from the relay setting development software to the • Relay Mapping - Each relay in the relay settings
relay setting database every time that an engineer calculates database must be mapped to the correct terminal in the
a relay setting with the relay setting development software. model and appropriate relay elements created for later
In the end, this reduces the potential for human error and settings population by the bulk import.
saves substantial time for the protection engineer tasked with The effort required for translation scripting is notable but
maintaining PRC-023 compliance. They no longer manually inherently scalable, as effort is roughly proportional to the
enter data into a spreadsheet but instead simply review and number of hardware relay models present in the system. This
validate that the data in the settings database is correct. From work is described next in Section IV-A.
the database, aggregate reports can be generated to export the In contrast, relay mapping for existing relays was at a scale
data and help prove compliance with this NERC standard. that Oncor deemed the effort too great a burden to distribute
2) Loadability Limit Calculations: The next benefit from among its engineering staff. Indeed, the magnitude of the task
the RSD/RSC API was the automated calculation and database is much greater than translation scripting as it must be done
storage of the relay loadability limit. This information is for every transmission relay in the entire power system. This
shared with the facility ratings teams to ensure that the is a common problem faced by large utilities that must be
relay’s load limit is taken into consideration for each terminal addressed so that the model accuracy requirements of PRC-
across the entire transmission system. This was accomplished 027 R1-1 can be met with a reasonable amount of engineering
performing similar steps as for PRC-023. However, the rating resources. While new and changed relays in the future can
limit can now be automatically included in an email sent to the be managed by incremental model maintenance defined in
facility rating teams when the relay settings are issued from the new settings development process, the challenge of relays
the relay setting database and have it. This again reduces time already in service required a new solution.
spent by the engineer in both calculations and transmitting We address the relay mapping challenge with a novel
data to the appropriate staff. automated approach that leverages the intersection of data

7
made available by interoperability between the relay settings stein algorithm for example, the score was reduced for
database, short circuit model, and settings calculator. This every incorrect character in approximate name matching.
algorithm is described in Section IV-B followed by subsequent Unlike most of the interoperability described in this paper,
verification efforts in Section IV-C. the implementation of this algorithm required tight, simul-
taneous integration of all three system protection software
A. Settings Translation Scripting
applications, as the algorithm relies on the combined capabil-
During the project, we developed settings translation scripts ities of the short circuit model, relay settings calculator, and
that are tailored both to Oncor’s relay setting standards and the relay settings database. The ability to automatically draw on
hardware relay models used by the company. These custom data stored or computed in all of them allowed us to quickly
scripts were designed to enable the import of relay settings create a solution which also used existing libraries for regular
without any engineer input, a necessary requirement for bulk expression transformations and approximate string matching.
automation. Even used in an ad-hoc manner without relay Our program implementing this algorithm generates a
mapping, the scripts allowed accurate modeling to be achieved spreadsheet summarizing mappings and associated confidence
much faster than previously possible. scores. The spreadsheet guided verification efforts and pro-
Validation of the translation scripting occurred by having vided a place to record comments for use by the team during
engineers manually map relays for new projects for a period the process. Once the mappings were complete, relays were
of time and verify the results of the complete relay linking created using the APIs of the short circuit model.
process for the small number of relays required. Verification While the algorithm was not able to match all cases,
included not just translated relay settings but also that the we achieved an overall 91.6% success rate with this initial
correct hardware relay model had been detected. After suffi- implementation in the project (see Table III), dramatically
cient testing for correctness as well as estimating the expected reducing the level of manual effort required by the team by
reduction of effort for bulk linking of the entire grid, work an order of magnitude.
began on an automated means of relay mapping.
C. Verification of Relay Linking
B. Automated Linking Algorithm
In order to verify the relay mapping and overall linking
Given the challenges described above, we developed a
effort, we divided the system up by work center for review
novel, automated approach to relay mapping. Briefly, our
and corrections as needed. For one of the work centers, we
mapping algorithm proceeds in the following steps which are
exhaustively verified that every relay was matched to the
also depicted in Figure 5:
correct item in the settings database. We did this by manually
1) Given a list of relays to map, extract unique database checking the database item that each relay was linked to
locations to map to short circuit model terminals. and verifying that the link pointed to the appropriate in-
2) Based on utility specific naming conventions and short service settings. Any relays that were mismatched or failed
circuit model information, use a collection of matching to match we linked by hand. This procedure allowed us to
techniques successively applied to attempt database to identify some corrections needed in the model and settings
model mappings. Example techniques include: database such as identifying stations that were not created in
• Regular expression transformations [10] such as the relay database at the time, name differences as stations
vowel removal or other shortening techniques used were upgraded, or voltage levels that had changed. We also
by engineers to meet varying length restrictions of identified a number of setting files missing from the database.
the relay database or short circuit model. This check demonstrated that the algorithm could correctly
• Approximate string matching [14] to account for identify the appropriate mapping in most cases as well as
typing errors or other naming anomalies. For this provide accurate suggestions for list of relays to verify or link
work, we used a modified Levenshtein Distance manually.
algorithm [12]. We ran the linking algorithm on the remaining work centers,
• Short circuit model topology analysis present in the continuing to make any corrections needed by hand. The whole
relay settings calculator. For example, Oncor’s relay process took about a month (interleaved with other, ongoing
database naming convention used breaker numbers work), yielding a verified model with a consistent relay naming
which were correlated to terminals via breaker as- convention and all transmission relays linked to the in service
sociations in the model. settings from the settings database. Table III demonstrates the
• Automated recalculation of relay settings such as performance of the linking algorithm by showing the number
primary line impedances and zone reaches using the of relays that were linked automatically and the number of
settings calculator and short circuit model that are relays that were linked by hand for each work center. It
then compared to candidate relay’s settings stored also shows the unexpected benefit of the algorithm’s results
in the relay database. to identify model discrepancies, that once resolved further
3) For locations matched above, assign confidence scores increased the accuracy of both the short circuit model and
based on technique that was used. Taking the Levench- data in the relay settings database.

8
 Fig. 5. Automated Linking Algorithm Overview.

TABLE III
L INK A LGORITHM A NALYSIS

Work Center Relays linked Relays that Relays verified Total number Percent linked Percent linked
automatically needed or linked by of relays automatically automatically
verification hand (adjusted
due to model for model
discrepancies discrepancies)
#1 253 112 20 385 65.7% 94.8%
#2 88 45 5 138 63.8% 96.4%
#3 102 28 19 149 68.5% 87.2%
#4 81 98 21 200 40.5% 89.5%
#5 299 258 29 586 51.0% 95.1%
#6 121 62 11 194 62.4% 94.3%
#7 249 88 26 363 68.6% 92.8%
#8 99 126 3 228 43.4% 98.7%
#9 146 47 5 198 73.7% 97.5%
#10 46 70 17 133 34.6% 87.2%
#11 51 71 4 126 40.5% 96.8%
#12 39 42 7 88 44.3% 92.0%
#13 112 60 17 189 59.3% 91.0%
#14 76 18 8 102 74.5% 92.2%
#15 32 22 9 63 50.8% 85.7%
#16 36 36 15 87 41.4% 82.8%
#17 59 45 12 116 50.9% 89.7%
#18 40 55 24 119 33.6% 79.8%
#19 46 30 2 78 59.0% 97.4%
#20 31 30 6 67 46.3% 91.0%
System Total 2006 1343 260 3609 53.6% 91.6%

After this effort, the relay settings development process A. Data Analytics
is significantly streamlined, as a settings engineer no longer
needs to have the database and the model opened at the We learned through these efforts that engineers spent sig-
same time nor manually ensure data consistency. Settings are nificant time performing tasks that could automated. This has
now retrieved with a button click within the model, reducing led to more efficiency for the relay setting engineers already
engineers’ time spent updating the model prior to creating new and even more gain is expected. Depicted in Figure 6 are the
settings. total counts for each status of the process prior to switching
to the new process. Figure 7 shows the counts for each status
V. L ESSONS L EARNED AND C HALLENGES after the new process was implemented. It can be determined
from these graphs that the new process clearly identifies the
When accounting for initial deployment of software solu- work being performed in each status and depicts the actual
tions, development of the new process, and the integration amount of work being performed. The efficiency gains of the
efforts described in this paper, this was a multi-year effort new process can be seen by the increase in overall counts of
involving numerous engineers. In this section, we describe a the statuses from the relay setting database.
few lessons learned and challenges faced. A challenge prior to the new process was trying to under-

9
 relay settings that were cancelled. With the new process
requiring that the workflow be maintained, it is now necessary
for the engineers to cancel the relay setting row and to make
a new one if modifications are made to what was previously
produced. These settings were mostly cancelled due to errors
that were found after the relay settings were issued and mostly
due to factors outside of the relay setting process.
Comparing the number of cancelled rows between the
previous process in Figure 6 and the new process in Figure 7
shows another benefit of this process; the ability to identify
the times that a relay setting had to be redone. Simple errors
can be identified by a change backwards in a singular step
in the process and major errors can be identified by a large
change in steps backwards in the process. These errors can be
seen below in Figure 8 which shows the status of the relay
setting in the database before it was moved to a prior status.
Therefore, the Peer Review status having the most amount of
changes to a prior status is expected, as this is where the errors
Fig. 6. Process Status Counts Prior to New Process should be found. The relays settings in Final Review before
being reverted could have either been a minor error or a major
error, depending on the prior status that they were moved to.
More analysis into these errors will be beneficial to better
understand the root cause in order to attempt error reduction.

Fig. 8. Relay Settings Process Success.

Fig. 7. Process Status With New Process

B. Field Support
Another important benefit from this project is reduced time
spent answering questions from the field. A necessary part of
stand how long it took to complete a relay setting project from the setting engineer’s job is to work with the technicians over
start to finish. As mentioned before, this challenge led to the the phone while settings are installed if anything unexpected
measured requirement (Section II-A) to capture the time spent comes up. Reducing calls due to preventable issues allows
performing relay setting work. This capture of the time can relay setters to spend more time on other tasks. Frequently,
be seen between the two graphs in Figure 6 and Figure 7. technicians will have questions regarding discrepancies be-
The setting statuses in the old process were mainly used to tween the download file and settings sheet or the test points.
track the settings from the review step through the release to We anticipate the new workflow will reduce field questions,
the field technician. This resulted in very little time difference due to the more defined review process as well as automation
shown between the steps as shown by the almost even counts of several steps.
for each step. Now that the new process covers the entirety of There are two separate review processes that are tracked
the project, the variation in work in a year can start to be seen through the settings database. Each setting must go through
in Figure 6 with the more detailed steps capturing the actual both review processes, ensuring that each setting is looked at
date of action. by at least two people, before it is issued to the field. Review
Another challenge was to find when errors were made and job aids assist the relay setters in thoroughly checking the
their impact. One way that errors can be found is to observe settings. The improved review process means that errors are

10
caught before issuing settings, reducing required interactions VII. F UTURE W ORK AND C ONCLUSIONS
with field technicians.
As discussed in Section III-D1, we automated the creation Oncor has already seen significant benefits outlined in the
of relay download files as well as testing points. Previously, paper both from the new settings development process as well
relay settings were manually typed into the download file and as the software packages and their interoperability that have
used the short circuit model to simulate faults and generate been combined to create an efficient, automated workfow that
testing points to ensure that the protection scheme as well as promotes settings accuracy. Additionally, our collaboration
settings are working as intended. Both of these provided op- during thse software integration efforts yielded several avenues
portunities for incorrect data to be provided to the technicians for exploration to further improve system protection activities:
in the field, and their deployment will reduce time spent on • Extended Data Analytics. - We continue to evolve our
the phone troubleshooting these types of errors. data analytics around the relay setting process with the
goal to better evaluate the relay setters work load. The
C. Challenges ability to pair past project information with projected
Much of this effort occurred during the early stages of project data to forecast the amount of relay setting work
the pandemic. Deployment and training of the relay settings will be very beneficial and allow us to more directly
calculator software was unexpectedly performed completely measure the impact of automation and interoperability
remote. Another remote challenge was faced by the settings efforts. With the process now showing the total time it
engineer tasked with development of the Excel workbook, who takes to complete a settings project, the goal is to better
was a relatively new engineer and had a basic understanding of understand the cost of full time employees completing
Oncor relaying. The setter learned the complexities of Oncor the work compared to contract resources. This will also
philosophy in a challenging environment, as questions could provide the ability to see relay setting workload over time
not be addressed in person as usual in the remote environment. rather than just a total count completed.
Debugging issues in a combined software solution was also • Automation Performance and Responsive Improve-
challenging, as was working with newly developed APIs for ments. One of the benefits of software interoperability is
interoperablity. It was often not clear which of the three vendor the ability to combine their interactions in a rich graphical
packages was causing the issue or if the issue was in fact user interface which the engineer can use to drive the
caused by an unknown security process or access policy on process. As is common across domains, an increase in
the deployed machines. All parties worked together not only speed (e.g., short circuit calculations or report generation)
to ensure the success of this project but also to bring greater enables a higher level of interaction with users, allowing
stability to the program interfaces. them to use the software in ways previously not possible.
We intend to further increase the performance of our inte-
VI. W IDE A REA C OORDINATION grated software solutions and explore the new interactions
PRC-027 R EQUIREMENT 2 this unlocks with engineers.
After the integrated process was deployed for use by settings • Autonomous Verification. Our short circuit model soft-
engineers, we turned interoperability efforts to wide area ware will soon store the model in a repository with every
coordination. An initial version of the solution described here change tracked akin to software source code revision
is currently being tested within Oncor. An overview of the control [15]. Extending this comparison, we envision
solution’s workflow is depicted in Figure 9. the system protection equivalent of continuous integra-
The relay settings calculator drives the coordination process, tion [16], with autonomous checks that verify expected
after model verification has been performed in a similar invariants, such as proper relay coordination, are not
manner as with the PRC-027 R1 process. Next, coordination violated by new settings or other model changes. This
criteria (e.g., allowable zone reach ranges, minimum CTI, will allow common sources of errors to be disccovered
etc) are specified and coordination is run on all relays in at earlier stages of the setting development process.
the area of study, utilizing the capabilities of the short circuit • Formal Mathematical Modeling of Protection Stan-
model. Violations are displayed in a filterable dashboard and dards. Building upon our foundations in settings automa-
can be interactively resolved in the settings calculator with tion [1], [2] and recent work in wide area coordination
verification occurring in the short circuit model. Finally, a bulk auto tuning [3], [4], we continue development of formal
transfer of relay settings updates and associated reports are mathematical specifications of fundamental problems in
sent to affected relays’ entries in the settings database. system protection. This effort is significantly aided by
One capability added to the setting calculator’s coordination interoperability, as it allows us to drive simulations to cre-
model as part of this project is the ability to compute an ate the formal models and subsequently invoke numerical
arbitrary symbolic mathematical expression using each relay’s solver libraries from within the same application.
settings, similar to capabilities used to implement the settings • Calculation Sheet Refinement. - We continue to add
philosophy template for the R1 process. The immediate use features to the line settings spreadsheets based on user
was for the wide area recalculation of loadability calculations, feedback. We are also developing excel based settings
though we anticipate much wider user in the future. sheets for all of our panels and relays to take advantage of

11
 Fig. 9. Wide Area Coordination Software Process Workflow

the features in excel that process automation. The lessons [14] G. Navarro. “A Guided Tour to Approximate String Matching.” ACM
learned from this project are vital in these future efforts. Computing Surveys. Association for Computing Machinery, New York,
NY, March 2001, pp.31–88.
We believe that the future is bright for innovation in system [15] J.Loeliger, Jon. “Version Control with Git: Powerful Tools and Tech-
protection software and automation. With software interoper- niques for Collaborative Software Development”. O’Reilly Media, Inc.,
2009.
ability, sophisticated solutions that assist throughout the entire [16] P. M. Duvall. “Continuous Integration. Improving Software Quality and
relay settings process promise to improve the productivity of Reducing Risk.” Addison-Wesley, 2007.
engineers and increase the reliability of the power grid by
reducing the chance for errors and providing the tools to move VIII. AUTHOR B IOGRAPHIES
forward despite growing complexity. Jared Gurley graduated from Texas Tech University with
a B.S. degree in Electrical Engineering in 2009. He works
R EFERENCES for Oncor as a manager in the relay setting group and is a
[1] M.J. Boecker, G.T. Corpuz, J. Perez, and L. Hankins. “Novel approach to registered professional engineer in Texas. He is responsible
relay setting development.” 2017 70th Annual Conference for Protective for managing a small team of engineers, overseeing contract
Relay Engineers (CPRE). IEEE, 2017. relay setting resources, and working with software to improve
[2] C. Thomas, J. Perez, L. Hankins, and H. Tribur. “An innovative and au-
tomated solution for NERC PRC-027-1 compliance.” 2018 71st Annual efficiency for the relay setting team. Jared previously worked
Conference for Protective Relay Engineers (CPRE). IEEE, 2018. in Transmission Operations and as a relay setting engineer.
[3] N. Thomas, L. Hankins and J. Perez, “Simplifying Wide Area Coor- Luci Hays earned B.S. and M.S. degrees in Electrical and
dination of Directional Time Overcurrent Relays Using Automatic Set-
tings Selection.” 2019 72nd Conference for Protective Relay Engineers Computer engineering from Baylor University in 2016 and
(CPRE), College Station, TX, USA, 2019. 2018, respectively. She works as a system protection engineer
[4] N. Thomas, L. Hankins and J. Perez, “Auto-tuned Solution to Wide for Oncor in the relay standards and studies group. She helps
Area Coordination Issues of Distance and Directional Time Overcurrent
Relay Settings.” 2021 74th Conference for Protective Relay Engineers design and develop standards, maintains existing standard
(CPRE), College Station, TX, USA, 2021. setting templates and drawings, and assists in documenting
[5] “PRC-027-1: Coordination of Protection Systems for Performance Dur- philosophies and procedures. She previously worked as a relay
ing Faults.” (2015, November). Retrieved February 22, 2018, from North
American Electric Reliability Corporation.
setting engineer.
[6] ASPEN Oneliner. https://www.aspeninc.com/web/software/oneliner. David Ridley earned his B.S. and M.S. degrees in Electrical
[7] ISO/IEC. (2020). “ISO International Standard ISO/IEC 14882:2020(E), and Computer engineering from Baylor University in 2017
Programming Language C++.” [Working draft]. Geneva, Switzerland:
International Organization for Standardization (ISO). Retrieved from
and 2019, respectively. He joined Oncor’s system protection
https://isocpp.org/std/the-standard group in 2019 as a relay setter. He works to create settings
[8] “The boost graph library: user guide and reference manual.” Addison- and troubleshoot issues with field technicians. He is part of a
Wesley Longman Publishing Co., Inc., USA, 2002.
team to improve engineer efficiency using automation.
[9] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein. 2009.
“Introduction to Algorithms, Third Edition.” The MIT Press, 2009. Zachary Austin graduated from Texas A&M University in
[10] A. V. Aho. “Algorithms for finding patterns in strings, Handbook of 2013 with his B.S. in Electrical Engineering. He is a registered
theoretical computer science (vol. A): algorithms and complexity.” MIT professional engineer in the state of Texas. He joined Oncor
Press, Cambridge, MA, 1991.
[11] Doble Enoserv PowerBase. https://www.doble.com/product/powerbase. Electric Delivery in 2013 and is currently the PowerBase
[12] V. I. Levenshtein. “Binary Codes Capable of Correcting Deletions, Programs Manager in the System Protection group. In this
Insertions and Reversals.” Soviet Physics Doklady, Vol. 10, February role he manages a team in the development, maintenance, and
1966, p.707.
[13] Setting Automation Relay Assistant. https://synchrogrid.com/sara- improvement of PowerBase at Oncor. Previously he spent six
software. years as a relay setting engineer.

12
 Nathan Thomas earned B.S. and Ph.D. degrees from Texas
A&M University in 1999 and 2012, respectively, both in
Computer Science. He has an extensive background in high
performance computing for large-scale engineering and scien-
tific applications. He is also interested in machine learning and
how it can be used to maximize system performance. Nathan
cofounded and leads development at SynchroSoft, the software
and automation division of SynchroGrid.
Luke Hankins graduated from Texas A&M University with
a B.S. degree in Electrical Engineering in 2015. He is a regis-
tered professional engineer in the state of Texas and works for
SynchroGrid in relay setting automation. He writes software
applications in various programming languages, focusing on
simplifying relay setting development. Luke also performs
relay settings verification and mis-operation analysis.
Alberto Borgnino graduated from the University of Genoa
with a M.S. Degree in Electrical Power Engineering in 1993.
He is a protection engineer with significant software develop-
ment experience. He has worked creating relay setting auto-
matic coordination systems and also has extensive experience
creating relay models in short circuit simulators.
Joe Perez received his B.S. degree in Electrical Engineering
from Texas A&M University in 2003. He is the author of many
relay application notes and technical papers at WPRC, Texas
A&M, and Georgia Tech Relay Conferences. Joe is the owner
of SynchroGrid, a registered professional engineer in the state
of Texas and a member of PSRC, IEEE, and PES.

13