Data Quality Control

Process
For Section 1600 Data Request Applications

September 2018

NERC | Report Title | Report Date

I
Table of Contents
Preface ....................................................................................................................................................................... iii
Introduction ............................................................................................................................................................... iv
Background ............................................................................................................................................................ iv
Approach................................................................................................................................................................ iv
Types of Quality Controls....................................................................................................................................... iv
Data Attributes ...................................................................................................................................................... iv
Roles and Accountabilities ..................................................................................................................................... iv
Chapter 1: Data Types and Data Preparation.............................................................................................................1
Data Requirements .................................................................................................................................................1
Communication and Training..................................................................................................................................2
Data Preparation.....................................................................................................................................................3
Data Access .............................................................................................................................................................4
Reporting ................................................................................................................................................................5
Chapter 2: Reporting Oversight ..................................................................................................................................7
Regional Entity Oversight........................................................................................................................................7
NERC Oversight .......................................................................................................................................................8
Late Reporting and Data Updates...........................................................................................................................9
Appendix A: Application References ....................................................................................................................... 11
Reporting Templates............................................................................................................................................ 11

NERC | Data Quality Control Process | September 2018

ii
Preface

The vision for the Electric Reliability Organization (ERO) Enterprise, which is comprised of the North American
Electric Reliability Corporation (NERC) and the seven Regional Entities (REs), is a highly reliable and secure North
American bulk power system (BPS). Our mission is to assure the effective and efficient reduction of risks to the
reliability and security of the grid.

The North American BPS is divided into seven RE boundaries as shown in the map and corresponding table below.
The multicolored area denotes overlap as some load-serving entities participate in one Region while associated
Transmission Owners/Operators participate in another.

FRCC Florida Reliability Coordinating Council

MRO Midwest Reliability Organization
NPCC Northeast Power Coordinating Council
RF ReliabilityFirst
SERC SERC Reliability Corporation
Texas RE Texas Reliability Entity
WECC Western Electricity Coordinating Council

NERC | Data Quality Control Process | September 2018

iii
Introduction

This Data Quality Control Process document describes the controls in place to ensure metric numeric accuracy for
Section 1600 data collection processes. The set of controls in place are designed to ensure that the Section 1600
application data are complete, accurate and timely to support analysis and compute metrics results that are
credible for assessing the reliability of the bulk electric system.

Background
As part of its scope or work plan, the application Working Group has been tasked by the Planning Committee with
developing a data quality control process that identifies the data collection process, data attributes, data controls,
and roles of the various entities who report, review, or analyze the data.

Approach
A risk-based approach was used to identify controls necessary to achieve the objective. A control framework is a
data structure that organizes and categorizes an organization's internal controls, which are practices and
procedures established to create business value and minimize risk. Risks and controls have been identified in each
of the high level process phases of reporting, compiling, analyzing and utilizing the application data.

Types of Quality Controls1

 Directive – designed to establish desired outcomes
 Preventive – designed to avoid an unintended event or result at the time of initial occurrence
 Detective – designed to discover an unintended event or result after the initial processing has occurred
but before the ultimate objective has concluded
 Corrective – designed to mitigate damage resulting from data errors

Data Attributes
The data attributes of completeness, accuracy and timeliness, are critical to the value of data collected. Within
each data type, this section identifies types of controls to address these data attributes.

Roles and Accountabilities

It is important that the various stakeholders involved in the reporting, collection and use of the application data
understand their respective roles and accountabilities. Roles and accountabilities are embedded in the list of
controls included within this section. The roles involved in collection of the application data, along with their
primary accountabilities pertaining to data quality include:

NERC:
 Owner of the application data
 Provides access to definitions and data reporting requirements
 Communicates reporting application updates with all stakeholders involved in the application data
reporting
 Provides application and subject-matter support to registered entities and REs

1Internal Control – Integrated Framework, Committee of Sponsoring organizations of the Treadway Commission (COSO), December 2011,
page 80

NERC | Data Quality Control Process | September 2018

iv
 Introduction

 Maintains and publishes the reporting deadlines

 Reviews the application data
 Coordinates with REs for data collection and communication with reporting entities
 Develops processes to ensure accuracy and timeliness of the application data
 Maintains the reporting application, including modifications, validation rules, and user security levels

The application Working Group:

 Provides industry stakeholder representation and subject-matter expertise to NERC
 Maintains definitions and data reporting requirements
 Makes recommendations to Performance Analysis Subcommittee and Planning Committee for updates to
data definitions and reporting requirements
 Suggests use of data for reports and metrics, application changes, and data quality controls
 Develops the application training materials
 Reviews analysis and reports that utilize the application data

Registered Entities:
 Primary suppliers of data reported
 May participate in the application Working Group
 Make and review recommendations of the application Working Group for updates to data definitions and
reporting requirements
 Suggests use of data for reports and metrics, application changes, and data quality controls
 Attend the application training offered by NERC

Regional Entities:
 Provide primary support of application and subject matter to registered entities and coordination with
NERC for additional support when necessary.
 Provide Regional expertise to NERC
 Review the application data reported by registered entities within the specific Region
 Follow up with registered entities with incomplete data reported for a reporting period
 May use data reported for Region-specific analysis and metrics

NERC | Data Quality Control Process | Draft – September 2018

v
Chapter 1: Data Types and Data Preparation

Data Requirements
The data requirements include the data elements, reporting periods, and reporting deadlines. These requirements
are described in a NERC Rules and Procedure (ROP) Section 1600 data request and corresponding data reporting
instructions. Minor changes to the Data Reporting Instructions are recommended to the Performance Analysis
Subcommittee and Planning Committee by the application Working Group. Major changes require a public review
period of the Section 1600 and corresponding data reporting instructions after approval by the Performance
Analysis Subcommittee and Planning Committee. The Board of Trustees approve all Section 1600 data requests,
including major changes.

Summary of Controls
Table 1.1: Summary of Controls – Data Requirements
Control Process or Documentation Completeness Accuracy Timeliness

Data Reporting Instructions, definitions,   

frequently asked questions and scenarios,
Directive
training materials, and other example
documents

Import templates or file formats and the   

Preventive application Working Group review and
recommendations

Public review, Performance Analysis  

Detective
Subcommittee, and Planning Committee

Corrective ERO Enterprise   

Risks to data quality

The application data requirements need to be clearly defined and made available to all data reporters to ensure
quality data. A lack of understanding of data requirements, formats, definition, or reporting periods may result in
inaccurate, incomplete, and lack of timely data, which impacts any analysis on the data.

Controls
Directive:
 Data Reporting Instructions specify the data required, the format of the data and reporting periods.
 Definitions provide explanations of the data elements to report.
 Frequently asked questions and scenarios outlining how to report common types of situations.
 Training materials provide applied, structured guidance on reporting procedures.

Preventive:
 Import template provides structure for entering data for submission to reporting application.

NERC | Data Quality Control Process | September 2018

1
 Chapter 1: Data Types and Data Preparation

 The application Working Group evaluates reporting issues or gaps and makes recommendations to
improve data collection or data quality.

Detective:
 Oversight of data reporting requirements from industry (public comment), Performance Analysis
Subcommittee and Planning Committee.

Corrective:
 The ERO Enterprise works with entities and specific data to resolve reporting errors.

Communication and Training

Various communication methods are used to provide data reporters with information about required data,
reporting deadlines, training opportunities and materials, and documentation to support the reporting process
for the application.

Summary of Controls
Table 1.2: Summary of Controls – Communication and Training
Control Process or Documentation Completeness Accuracy Timeliness

Data Reporting Instructions, definitions,   

frequently asked questions and scenarios,
Directive training materials, other example documents,
and reporting deadlines posted on NERC’s
website.

Import templates, the application Working   

Preventive Group, training sessions, Regional Entity, and
dedicated application E-mail.

Corrective ERO Enterprise   

Risks to data quality

A lack of access to materials and resources results in a lack of understanding of data requirements, formats,
definition, or reporting periods that may result in inaccurate, incomplete, and lack of timely data, which impacts
any analysis on the data.

Controls
Directive:
 Data Reporting Instructions specify the data required, the format of the data and reporting periods.
 Definitions provide explanations of the data elements to report.
 Frequently asked questions and scenarios outlining how to report common types of situations.
 Training materials provide applied, structured guidance on reporting procedures.

NERC | Data Quality Control Process | Draft – September 2018

2
 Chapter 1: Data Types and Data Preparation

 Reporting deadlines posted on the application-specific web page provide easily accessible information for
timely reporting.

Preventive:
 Import template provides structure for entering data for submission to reporting application.
 Training sessions provide opportunity for interaction with subject-matter experts.
 Assigned RE to provide personal training and contact registered entities about approaching or missed
reporting deadlines.
 Dedicated application E-mail allows users to communicate to a distribution list of NERC staff supporting
the application.
 NERC staff communicate with technical experts from working group to answer highly technical questions
for reporting frequently asked questions and scenarios.

Corrective:
 The ERO Enterprise works with entities and specific data to resolve reporting errors.

Data Preparation
The registered entities use the application Data Reporting Instructions and associated templates, as applicable, to
guide the preparation of data for submission to the application.

Summary of Controls
Table 1.3: Summary of Controls – Data Preparation
Control Process or Documentation Completeness Accuracy Timeliness

Data Reporting Instructions, definitions,   

frequently asked questions and scenarios,
Directive
training materials, and other example
documents.

Preventive Import templates or file formats   

Validations within the reporting application or  

Detective template used by the registered entity or RE
before submittal to NERC

Validations within the reporting application

used by the ERO Enterprise after data is  
Corrective
submitted to NERC in order to detect accepted
validation errors

Risks to data quality

Registered entities reporting to the application prepare the data in the format or template using the information
provided in the application Data Reporting Instructions and supporting materials. A lack of understanding of data
requirements, formats, definition, or reporting periods may result in inaccurate, incomplete, and lack of timely
data, which impacts any analysis on the data.

NERC | Data Quality Control Process | Draft – September 2018

3
 Chapter 1: Data Types and Data Preparation

Controls
Directive:
 Data Reporting Instructions specify the data required, the format of the data and reporting periods.
 Definitions provide explanations of the data elements to report.
 Frequently asked questions and scenarios outline examples of how to report common types of situations.
 Training materials provide applied, structured guidance on reporting procedures.

Preventive:
 Import templates or defined file formats provides structure for preparing data for submission to reporting
application.

Detective:
 Validations with the reporting application or template test validity of specific types of data errors.
 Validations may present warnings and accept the data or completely reject the submission until errors
are corrected.
 Validations with the reporting application alert the RE to specific types of data errors and may present
warnings and accept the data or completely reject the submission until errors are corrected.

Corrective:
 Validations within the reporting application used by the ERO Enterprise after data is submitted to NERC
to detect accepted validation errors

Data Access
Users of the application must to be authorized to submit data for their entity. A vetting process or digital certificate
may be required to grant access to a user. The application must not permit a user from one entity to have access
to another entity’s data without authorization from the entity.

Summary of Controls
Table 1.4: Summary of Controls – Data Access
Control Process or Documentation Completeness Accuracy Timeliness

Training materials and other supporting  

Directive materials describe requirements and process
for access.

User setup requires digital certificate or entity  

Preventive
approval for access; role-based functionality.

Digital certificate requires coordination with  

Detective third party; user access and transactions are
tracked in the application.

Security in the reporting application prevent

Corrective
unauthorized access to another entity’s data.  

NERC | Data Quality Control Process | Draft – September 2018

4
 Chapter 1: Data Types and Data Preparation

Risks to data quality

Registered entity access to another entity’s data could result in compromised data through data manipulation or
access to confidential business operating information.

Controls
Directive:
 Training materials and other supporting documents provide guidance on access requirements.

Preventive:
 Access to the reporting application may require a digital certificate to be issued to the user. This registers
the user and company to a specific computer.
 Some reporting applications require a NERC account and a vetting process to obtain access to the
reporting application.
 NERC sets up the users in the applications that require a digital certificate.
 Where a vetting process occurs, the reporting entity vets the user requesting access.
 Functionality is limited based on permissions.

Detective:
 Users must obtain a digital certificate from a third party for some web-based reporting applications.
 User access and transactions are tracked within the application.

Corrective:
 Security in the reporting application prevent unauthorized access to another entity’s data.

Reporting
Registered entity users with permission to report, use import templates or defined file formats to report data
based on a schedule. The reporting schedule permits the data to be collected on an interval basis that has been
defined in the Section 1600 data request. Reporting deadlines may vary by application and are made available to
the registered entities through information in the reporting application or the NERC web page specific to that data
reporting application.

Summary of Controls
Table 1.5: Summary of Controls – Reporting
Control Process or Documentation Completeness Accuracy Timeliness

Data Reporting Instructions, definitions,   

frequently asked questions and scenarios,
Directive training materials, other example documents,
and reporting deadlines posted on NERC’s
website.

Import templates, E-mail reminders, and   

Preventive
reporting checklists

NERC | Data Quality Control Process | Draft – September 2018

5
 Chapter 1: Data Types and Data Preparation

Table 1.5: Summary of Controls – Reporting

Control Process or Documentation Completeness Accuracy Timeliness

Validations within the reporting application or   

Detective
template and reporting checklists

Corrective ERO Enterprise

 

Risks to data quality

A lack of understanding of data requirements, formats, definition, or reporting periods may result in inaccurate,
incomplete, and lack of timely data, which impacts any analysis on the data.

Controls
Directive:
 Data Reporting Instructions specify the data required, the format of the data and reporting periods.
 Definitions provide explanations of the data elements to report.
 Frequently asked questions and scenarios outlining examples of how to report common types of
situations.
 Training materials provide applied, structured guidance on reporting procedures.

Preventive:
 Import template provides structure for entering data for submission to reporting application.
 Automated reminders are sent to registered entity users to remind them of upcoming reporting
obligations. These are typically sent at various intervals up to the reporting deadline, such as two weeks
prior, one week prior, and one day prior.
 Reporting checklists provide registered entities with a list of required data and the reporting status. When
all data are provided, the checklist displays as complete.

Detective:
 Validations within the reporting application test the validity of the reported data and prevent a checklist
from being reported as complete when there are issues with the data that was submitted.
 Entities may enter comments on the checklist to alert the RE or NERC to any issues with the data for a
given period.

Corrective:
 The ERO Enterprise works with entities and specific data to resolve reporting errors.

NERC | Data Quality Control Process | Draft – September 2018

6
Chapter 2: Reporting Oversight

Each Section 1600 data request specifies the reporting interval and reporting period deadline for registered
entities. This provides time after the end of the reporting period for the registered entities to gather and review
data before submission. REs have an additional period following the registered entities’ reporting period deadline.
This chapter describes the controls for coordinated oversight by NERC and REs. REs may have additional internal
controls not included in this document.

Regional Entity Oversight

REs monitor reporting progress of all entities in the Region through the Regional checklist and may review the
detail of the checklist of any entity in the Region. Reporting checklist information may be exported to Excel.

Summary of Controls
Table 2.1: Summary of Controls – Regional Oversight
Control Process or Documentation Completeness Accuracy Timeliness

Regions use the Data Reporting Instructions,   

definitions, frequently asked questions and
Directive scenarios, training materials, and other
example documents to inform the registered
entities of their reporting obligations.

Regional checklists, Regional training provided   

Preventive
to registered entities

Validations within the reporting application or   

Detective template and reporting checklists
Reporting verification process training for REs

Regional review period and ERO Performance

Corrective   
Analysis group

Risks to data quality

A lack of understanding of data requirements, formats, definition, or reporting periods may result in inaccurate,
incomplete, and lack of timely data, which impacts any analysis on the data.

Controls
Directive:
 Regions and registered entities have access to the Data Reporting Instructions, training materials and
other support materials available on NERC’s website.
 NERC’s reporting verification process training provides REs information related to the reporting timeline.
Each RE may have internal procedures for evaluating reported data and contacting entities that have not
completed their reporting for a period.

NERC | Data Quality Control Process | September 2018

7
 Chapter 2: Reporting Oversight

Preventive:
 Regions may provide one-on-one or other training support to registered entities to help them with
reporting issues or questions.
 REs may monitor reporting progress of all entities in the Region through the Regional checklist and may
review the detail of the checklist of any entity in the Region.
 NERC may monitor the reporting progress of entities at the individual or Regional level.

Detective:
 The reporting application provides periodic updates to notify the Regions of entities that have not
reported by the reporting deadline.
 NERC sends Regional checklists to the Regional application contact, showing the reporting status of each
registered entity for the reporting period.

Corrective:
 Regions have a review period in which to attempt to collect additional data from registered entities after
the reporting deadline.
 Regions have the ability to import data for a registered entity during the Regional review period.
 REs communicate with NERC through an ERO Performance Analysis group to identify and resolve data
reporting issues in their Region.

NERC Oversight
NERC provides user support and manages the applications that collect data from the registered entities. NERC
monitors the progress of data reporting through a summary checklist that includes the reporting status of the
entities or regions by reporting period. NERC supports the Regions to upload data provided by a registered entity
after the Regional review period closes. NERC monitors, updates, and reports quarterly dashboard on
completeness of reporting at the Regional level.

Summary of Controls
Table 2.2: Summary of Controls – NERC Oversight
Control Process or Documentation Completeness Accuracy Timeliness

User support to inform users of available   

Directive resources for user setup, data reporting, or
application updates.

Import templates and user support, e-mail   

Preventive
reminders to registered entities or REs

Use of application dashboard to notify users of  

Detective
any application issues

Regional review period and ERO Performance

Corrective   
Analysis group

NERC | Data Quality Control Process | Draft – September 2018

8
 Chapter 2: Reporting Oversight

Risks to data quality

A lack of understanding of data requirements, formats, definition, or reporting periods may result in inaccurate,
incomplete, and lack of timely data, which impacts any analysis on the data.

Controls
Directive:
 NERC guides users to applicable resources to address user support inquiries.
 NERC uses user lists to notify users of application issues or updates.

Preventive:
 NERC develops and maintains the import templates that provide the structure for entering data for
submission to reporting application.
 NERC sets up the reminder E-mails that are sent from each reporting application to reporting entities and
REs.

Detective:
 NERC uses the application dashboard (where applicable) to notify users of any application issues

Corrective:
 The ERO Enterprise works with entity/entities and specific data to resolve reporting errors.
 NERC communicates with REs through an ERO Performance Analysis group to identify and resolve data
reporting issues in their Region.

Late Reporting and Data Updates

Registered entities may still provide missing data or make corrections to data after the reporting deadline or the
close of the Regional review period. This may occur because the person responsible for reporting has left the
entity or updated information becomes available. The reporting application prevents the registered entity or the
RE from entering data after the specified deadlines. Data reported to NERC after the Regional review period is
imported by NERC staff, provided it is sent to NERC in the correct import format. When validation errors or
incomplete checklists occur, NERC staff work with the registered entity to resolve issues and complete the data
imports. A process to accept late data or updates after reporting periods are closed out by the application allows
NERC to maintain a more complete representative data set.

Summary of Controls
Table 2.3: Summary of Controls – Late Reporting
Control Process or Documentation Completeness Accuracy Timeliness

NERC Performance Analysis Late Reporting  

Directive
process

Preventive Import templates or file formats  

Validations within the reporting application or  

Detective
template and reporting checklists

NERC | Data Quality Control Process | Draft – September 2018

9
 Chapter 2: Reporting Oversight

Table 2.3: Summary of Controls – Late Reporting

Control Process or Documentation Completeness Accuracy Timeliness

ERO Enterprise coordination with registered

Corrective
entity  

Risks to data quality

Incomplete data impacts analysis and may result in sample sizes that are not representative of a given data set.

Controls
Directive:
 The NERC Performance Analysis Late Reporting process provides instructions to regions on accepting late
or updated data provided after the entity and Region periods have closed in the application
 Supporting materials provide REs with references to NERC’s authority to collect this data from registered
entities and templates for requesting late data.

Preventive:
 Import template provides structure for entering data for submission to reporting application. Entities are
required to provide data to NERC and REs via the import template to ensure that validations are applied
to incoming data.

Detective:
 Validations within the reporting application test the validity of the reported data and prevent a checklist
from being reported as complete when there are issues with the data that was submitted.

Corrective:
 ERO Enterprise works directly with the registered entity to resolve any validation errors resulting from a
late submission.

NERC | Data Quality Control Process | Draft – September 2018

10
Appendix A: Application References

Reporting Templates
The links below may be used to access the reporting template for each application.

Reporting Template Links

Table A.1: Reporting Template Links
Application NERC Web Page Link

Application reference page:

http://www.nerc.com/pa/RAPA/dads/Pages/default.aspx
Demand Response Availability
Data System (DADS) Bulk upload format and instructions:
http://www.nerc.com/pa/RAPA/dads/Documents/Data%20Reporting%2
0Instructions.pdf

Application reference page:

http://www.nerc.com/pa/RAPA/gads/Pages/default.aspx
Generating Availability Data
Systems (GADS) Data Reporting Instructions:
https://www.nerc.com/pa/RAPA/gads/Pages/Data%20Reporting%20Inst
ructions.aspx

Application reference page, Data Reporting Instructions (DRI), and

GADS - Wind reporting templates:
https://www.nerc.com/pa/RAPA/gads/Pages/GADS-Wind-DRI.aspx

Application reference page, Data Reporting Instructions (DRI), and

Misoperations Information Data reporting templates:
Analysis System (MIDAS)
http://www.nerc.com/pa/RAPA/Pages/Misoperations.aspx

Application reference page, Data Reporting Instructions (DRI), and

Transmission Availability Data reporting templates:
System (TADS)
http://www.nerc.com/pa/RAPA/tads/Pages/default.aspx

NERC | Data Quality Control Process | September 2018

11