Transforming Utility Networks: Large-scale Data Migrations with FME

Transforming Utility
Networks:
Large-scale Data
Migrations with FME
TRC Companies
The Peak of Data
and AI 2025

2025
The
Peak
of
Data
and
AI
Jeremy Conner
Senior GIS Consultant
TRC Companies, IGS
Daniella Mesler
Senior GIS Data Analyst
TRC Companies, IGS
Freddy Neal
Senior Data Architect
TRC Companies, IGS

1. Introduction
2. The Challenges
3. Our Solutions
4. Key Takeaways
Agenda

2025
The
Peak
of
Data
and
AI
TRC Companies & Intelligent
Grid Solutions

2025
The
Peak
of
Data
and
AI
Strategic Delivery Relationship: Our partnership with
Esri is cornerstone to delivering successful GIS projects
for utilities.
+
1st Utility Network go-live
globally @ National Grid in
2017 12
Utility Network
production go-lives in the
last four years
years as Esri
platinum partner
12
Dedicated GIS
professionals
400+ Utility Network projects in ﬂight
8
50+
System
Integration
projects
Industry proven methodology for Utility
Network POC’s & Implementations
#1

2025
The
Peak
of
Data
and
AI
Project Conditions
● One of the largest fully regulated utility companies.
● Already completed several migration cycles.
● Transforming on-prem legacy data from geometric network into Universal Pipeline Data
Model on SAP HANA.

2025
The
Peak
of
Data
and
AI
Large amount of data to migrate
● ~300 million features across gas, electric and gas GPS data.
● Large amounts of data processed in a linear fashion.
● Small editing freeze window.

2025
The
Peak
of
Data
and
AI
Writing Restrictions with UtilityNetwork
Dataset
● Loading data via Asset Package is time
prohibitive.
● Writing via REST is not a viable option.
● Overly broad error reporting using GP tools

2025
The
Peak
of
Data
and
AI
Data Originated from Disparate
Databases
Varying data sources needing to live under one roof.
● Geometric Network Database for each region
● GPS database for each region
● PODS Transmission
● M&R
● Two different EAMs

2025
The
Peak
of
Data
and
AI
Many Network Errors to Remediate
● In early data migration cycles, errors
totaled ~ 2 million.
● Excessive network errors inhibit the
ability to Enable Topology in the UN.
● Subnetworks rely upon clean
network connectivity along with
subnetwork controller and terminal
conﬁguration.

2025
The
Peak
of
Data
and
AI
FLOW – Automating What You
Automated
Automations allowed us to seamlessly thread together complex, multi-process workﬂows.
● Error Remediation
● Python script execution
● Webhooks
● Schema Mapping
● Loading
● SQL SPROC Calls

2025
The
Peak
of
Data
and
AI
Parallelism
● Our biggest contributor to solving long data loading times.
● Spread writing jobs over multiple servers with 10-15 engines each.
● Applicable to more than data loading.

2025
The
Peak
of
Data
and
AI
Staging Tables Utility Network
Transform Data
Maneuvering Around Writing Restrictions
● Averted branch version and GlobalID-related conﬂicts associated with parallel
writing.
● Performed remediation spatial processing.
● Executed Append GP tool from staging tables via python caller within automations.
• Schema Mapping
• New Features
• Error Remediation
• QAQC
• Appends
• Globalid Retention

2025
The
Peak
of
Data
and
AI
Handling Data from Disparate Locations and
Creating Net-new features
● Joined data from various silos to build out GIS features
for station asset
● Employed a “rubber stamp” approach for simpler
conﬁgurations.
● Developed custom looping transformer to search for
"optimal" placement location.

2025
The
Peak
of
Data
and
AI
Large Reduction in Network Errors
● Leveraged Flow Automations, increased engine usage, parallelism and
staging feature classes to dramatically reduce network errors.
● Used FME to build solutions for UN Errors:
○ Created proxy junctions for pipes of differing asset groups/types.
○ Performed Z-offsets for coincident features.
○ Connectivity Associations and logic to create new connectivity rules.
○ Enhanced attribution via spatial relationships of point & line features.
Pre-ER
2,000,000
Post-ER
200,000

2025
The
Peak
of
Data
and
AI
Scalable Parallelism Logic

2025
The
Peak
of
Data
and
AI
Employ Automations Often
Chain Them Together
Less Intervention
Centralize Parameters
Notiﬁcations

2025
The
Peak
of
Data
and
AI
TRC successfully scaled
data processing at all key
milestones of data model
transformation

2025
The
Peak
of
Data
and
AI
ThankYou
Freddy Neal & Daniella Mesler
TRC Companies
Fneal@trccompanies.com & Dmesler@trccompanies.com

Transforming Utility Networks: Large-scale Data Migrations with FME

More Related Content

Similar to Transforming Utility Networks: Large-scale Data Migrations with FME (20)

More from Safe Software (20)

Recently uploaded (20)

Transforming Utility Networks: Large-scale Data Migrations with FME