Module 3

History Data Storage and Retrieval

Section 1 – Data Retrieval 3-3
Section 2 – Historian Time Domain Extensions 3-9
Lab 3 – Query Examples 3-25
3-2 Module 3 – History Data Storage and Retrieval

Module Objectives
z Explain high-level data retrieval and changes to the Historian OLE-DB Provider
z Explain Historian SQL Query structures
z Use SQL examples to show query syntax and data retrieval options

Wonderware Training
 Section 1 – Data Retrieval 3-3

Section 1 – Data Retrieval

Section Objective
z Explain the Data Retrieval Subsystem and Features for the Historian

This section explains the Data Retrieval Subsystem and features for the Historian.

Data Retrieval Subsystem

The purpose of the retrieval subsystem is to receive SQL queries from clients, locate the
requested data, perform any necessary processing, and then return the results. For configuration
and event data, retrieval is made possible by normal SQL queries, because these types of data
are stored in normal SQL Server database tables. History data, however, must be retrieved from
the history blocks and presented to clients as if it were also stored in SQL Server tables. To
accomplish retrieval from both of these data repositories, the retrieval subsystem includes:
z An implementation of a SQL Server data provider, which determines whether the
requested data resides in normal SQL Server tables or in the history blocks.
z A low-level retrieval layer, which is responsible for extracting the requested data from the
history blocks and presenting to the InSQL OLE DB provider as "virtual" history tables.
z A set of SQL Server extensions, which are implemented as columns in the history tables.
You can use these extensions to specify the nature of the rowset that will be returned,
such as the number of rows returned, the resolution of the data, or the retrieval mode.
For more information on data storage, see the Online Help topic: Data Storage Subsystem.

Data Retrieval Features

Some of the main features of the data retrieval subsystem are:
z All tag types can be included in the same query, when retrieving from the History table.
You do not need to separate analog, discrete, and string tags into different queries. Any
combination of tags may be submitted in a query.
z Live data from the active image will be returned independent of the tag types in the list,
unless an ORDER BY clause is applied to the query.
z Both fixed length and variable length strings are supported.
z All internal time computation and manipulation is done using the Win32 FILETIME type.
The resolution of FILETIME is 100 nano-seconds.
z All times are handled internally as absolute time (UTC). Conversions to and from local
time are handled going in and out of retrieval so the external interface is local time.
z Non-realtime data is supported (for example, store-and-forward data or data imported
from a .CSV file).
z Retrieval of different versions is supported.

Wonderware System Platform Course - Part 2

3-4 Module 3 – History Data Storage and Retrieval

History Blocks: A SQL Server Remote Data Source

Remote data sources are simply data repositories that exist outside of a Microsoft SQL Server
database file (.MDF). Microsoft sometimes refers to these types of data sources as "non-local data
stores." In the case of the Historian, a remote data source is the set of history block files. All tag
data is stored in the history blocks. For more information on history blocks, see the Online Help
topic: History Blocks.
OLE DB technology can be used to access data in any remote data store. This access is
accomplished though a software component called an OLE DB provider.

About the InSQL OLE DB Provider

Object Linking and Embedding for Databases (OLE DB) is an application programming interface
(API) that allows COM-based client applications to access data that is not physically stored in the
Microsoft SQL Server to which they are connecting.
The benefit of using OLE DB is that it provides access to different types of data in a broader
manner. By using OLE DB, you can simultaneously access data from a variety of sources, such as
from a Microsoft SQL Server database, an Oracle database, and a Microsoft Access database. A
query that accesses data from multiple, dissimilar data sources such as these is called a
"heterogeneous query," with "heterogeneous" meaning "dissimilar." A heterogeneous query can
also be called a "distributed query," because the data can be distributed across various data
sources.
Microsoft SQL Server uses OLE DB to process heterogeneous queries and makes linking data
between the data sources much easier. Through OLE DB, Microsoft SQL Server supports
Transact-SQL queries against data stored in one or more SQL Server and heterogeneous
databases without any need for specialized gateway server applications.
The interface required to access data in a non-local data store (such as the Historian history
blocks) is provided by a "virtual" server, called an OLE DB provider. OLE DB providers allow you to
use the power of the Microsoft SQL Server query processor to make linking data stored in the
Microsoft SQL Server databases and from the history blocks much easier and more robust. Also,
the Historian OLE DB provider has a rich set of query capabilities.
The name of the Historian OLE DB provider is "INSQL." The INSQL OLE DB provider is installed
during Historian installation and then associated, or linked, with the Microsoft SQL Server. For
information on the syntax for linking the INSQL OLE DB provider, see the Online Help topic:
Linking the INSQL OLE DB Provider to the Microsoft SQL Server.
To access historical data using OLE DB, any COM-based client application must connect directly
to the Microsoft SQL Server and then specify to use the Historian OLE DB provider in the syntax of
the query.
When you execute a query and specify the Historian OLE DB provider in the syntax, the Microsoft
SQL Server parser will pass the appropriate parts of the data request to the Historian Server OLE
DB provider. The Historian OLE DB provider will then interface with the low-level retrieval layer to
locate the data store, extract the requested information, and return the data to the Microsoft SQL
Server as a rowset. Microsoft SQL Server will perform any other processing required on the data
and return the data to the client application as a result set and a set of output parameters, if
applicable.
The Historian OLE DB provider must be present on the server running Microsoft SQL Server. The
set of Transact-SQL operations that can be used to retrieve data in the history blocks depends on
the capabilities of the Historian OLE DB provider. The Historian OLE DB provider is SQL-92
compliant. For more information on OLE DB, see your Microsoft documentation.

Wonderware Training
 Section 1 – Data Retrieval 3-5

Extension (Remote) Tables for History Data

Some of the history tables are rowset representations that provide a means for handling acquired
plant data. These tables are not part of normal SQL Server functionality. A normal SQL Server
table stores data directly in the database's data device file (.mdf). An extension table, however,
presents data as if it were a real table, but it does not physically exist in the data device. An
extension table is a logical table that is populated from other types of data files; thus, the data is
stored "remotely" from SQL Server. In the case of the Historian, the data files are the history
blocks generated by the storage system.
Note Extension tables are also called remote tables.
Data access from the history blocks is made possible by SQL Server's OLE DB provider
technology. Client applications must connect directly to the Microsoft SQL Server and then specify
to use the Historian OLE DB provider in the syntax of the query.
The extension tables are:
z History (InSQL.Runtime.dbo.History)
z HistoryBlock (INSQL.Runtime.dbo.HistoryBlock)
z Live (INSQL.Runtime.dbo.Live)
z WideHistory (INSQL.Runtime.dbo.WideHistory)
The AnalogHistory, DiscreteHistory, StringHistory, AnalogLive, DiscreteLive, StringLive,
AnalogWideHistory, DiscreteWideHistory, and StringWideHistory tables are provided for backward
compatibility. For more information, see the Online Help topic: Backward Compatibility Entities.
The AnalogHistory, DiscreteHistory, StringHistory, and History tables are the only tables which are
updateable. The remaining tables are read-only.
For more information on the history extension tables, see the Online Help topic: History Tables.

Query Syntax for the Historian OLE DB Provider

The most common Historian query is a SELECT statement:
SELECT select_list
FROM table_source
WHERE search_condition
[ GROUP BY group_by_expression ]
[ HAVING search_condition ]
[ ORDER BY order_expression [ ASC | DESC ] ]
A WHERE clause is mandatory when issuing a SELECT query against any extension table except
HistoryBlock.
There are four variations for issuing a SELECT statement to the Historian OLE DB provider to
retrieve history data, which will be discussed next:
z Four-Part Naming Convention
z Historian OLE DB Provider View
z OPENQUERY Function
z OPENROWSET Function

Wonderware System Platform Course - Part 2

3-6 Module 3 – History Data Storage and Retrieval

Using the Four-Part Naming Convention

The linked server name is simply a name by which the Historian OLE DB provider is known to the
Microsoft SQL Server. In order for a query to be passed on to the Historian OLE DB provider, you
must specify the linked server name and the extension table name as part of a four-part naming
convention.
For example, this query specifies to retrieve data from the History extension table in the Historian
OLE DB provider:
SELECT * FROM INSQL.Runtime.dbo.History
WHERE TagName = 'SysTimeSec'
AND DateTime >= '2001-09-12 12:59:00'
AND DateTime <= '2001-09-12 13:00:00'
The four-part naming convention is described in the following table:

Part Name Description

linked_server Linked server name. By default, INSQL.
catalog Catalog in the OLE DB data source that contains the object from which you
want to retrieve data. For Microsoft SQL Server type databases, this is the
name of the database. To use the Historian OLE DB provider, the catalog
name will always be "Runtime."
schema Schema in the catalog that contains the object. For Microsoft SQL Server
type databases, this is the name of the login ID for accessing the data. To
use the Historian OLE DB provider, the catalog name will always be "dbo."
object_name Data object that the OLE DB provider can expose as a rowset. For the
Historian OLE DB provider, the object name is the name of the remote table
that contains the data you want to retrieve. For example, the History table.

In the case of four-part queries, SQL Server produces the statement that is sent to the Historian
OLE DB provider from the statement that the user executes. Sometimes this produced statement
is incorrect, too complex, or lacks portions of the WHERE clause required for the Historian OLE
DB provider to return data.
A typical error message when executing unsupported syntax is:
Server: Msg 7320, Level 16, State 2, Line 1
Could not execute query against OLE DB provider 'INSQL'.
[OLE/DB provider returned message: Historian did not receive a WHERE clause from SQL Server.
If one was specified, refer to the Historian OLE DB documentation]
For four-part queries against non-English SQL Servers running on non-English operating systems,
the default date format might differ from the English versions. For example, for a French or
German SQL Server running on the corresponding operating system, the date/time in a four-part
query must be:
yyyy-dd-mm hh:mm:ss.fff
For example:
2003-28-09 09:00:00.000

Wonderware Training
 Section 1 – Data Retrieval 3-7

Using a Historian OLE DB Provider View

Microsoft SQL Server views have been provided that will access each of the extension tables,
eliminating the need to type the four-part server name in the query. These views are named the
same as the provider table name. Note Backward compatibility views are named according to the
v_ProviderTableName convention.
For example:
SELECT * FROM History
WHERE TagName = 'SysTimeSec'
AND DateTime >= '2001-09-12 12:59:00'
AND DateTime <= '2001-09-12 13:00:00'

Using the OPENQUERY Function

You can use the linked server name in an OPENQUERY function to retrieve data from an
extension table. The OPENQUERY function is required to retrieve wide table data. For example:
SELECT * FROM OPENQUERY(INSQL, 'SELECT * FROM History
WHERE TagName = "SysTimeSec"
AND DateTime >= "2001-09-12 12:59:00"
AND DateTime <= "2001-09-12 13:00:00" ')
The following example retrieves data from a wide table:
SELECT * FROM OPENQUERY(INSQL, 'SELECT DateTime, SysTimeSec
FROM WideHistory
WHERE DateTime >= "2001-09-12 12:59:00"
AND DateTime <= "2001-09-12 13:00:00" ')
The OPENQUERY portion of the statement is treated as a table by SQL Server, and can also be
used in joins, views, and stored procedures. SQL Server sends the quoted statement, unchanged
and as a string, to the Historian OLE DB provider. Consequently, only the syntax that the Historian
OLE DB provider can parse is supported. Also, be sure that you do not exceed the 8000 character
limit for the statement. Consider the following example:
SELECT * FROM OpenQuery(INSQL, 'XYZ')
where "XYZ" is the statement to pass. You should be sure that the value of "XYZ" is not more than
8000 characters. This limit is most likely to cause a problem if you are querying many tags from a
wide table.
Also, you should supply the datetime in an OPENQUERY statement in the following format:
yyyy-mm-dd hh:mm:ss.fff
For example:
2001-01-01 09:00:00.000
You cannot use variables in an OPENQUERY statement. For more information, see the Online
Help topic: Using Variables with the Wide Table.

Using the OPENROWSET Function

The linked server name can be used as an input parameter to an OPENROWSET function. The
OPENROWSET function sends the OLE DB provider a command to execute. The returned rowset
can then be used as a table or view reference in a Transact-SQL statement. For example:
SELECT * FROM OPENROWSET('INSQL',' ', 'SELECT DateTime, Quality,
QualityDetail, Value
FROM History
WHERE TagName in ("SysTimeSec")
AND DateTime >= "2001-09-12 12:59:00"
AND DateTime <= "2001-09-12 13:00:00" ')

Wonderware System Platform Course - Part 2

3-8 Module 3 – History Data Storage and Retrieval

– Intentionally left blank –

Wonderware Training
 Section 2 – Historian Time Domain Extensions 3-9

Section 2 – Historian Time Domain Extensions

Section Objective
z Explain the Data Retrieval Subsystem and Features for the Historian

This section explains the Data Retrieval Subsystem and Features for the Historian.

Data Retrieval Methods

The SQL language is not always appropriate for handling time series data. The Historian adds
several extensions to the normal SQL language to make it possible to meaningfully query this type
of information. The methods of data retrieval can be grouped by function as follows:
z Time interpolation
z Wide Table Transformations
z Storage Independent Retrieval Mode
z Resolution Control
z Edge Detection
z Time Domain Extensions
Each of the above functions is described in the following pages. The various extensions can be
used together to achieve a wide range of effects. The extensions only apply to the live (real time)
and history tables.

Time Interpolation
This extension allows data to be retrieved at any point in time even if there is no row at that point in
time.
Why This is Needed
An unassisted database would return data from history for a specified point in time only if there
were an actual row with that time stamp. If there were no row data, complex SQL would be
required to determine the effective value.
How It Is Used
Nothing special is required to use Time Interpolation. This is inherent in the way the Historian
handles all queries to the extension data.
For example the queries:
SELECT Value
FROM InSQL.runtime.dbo.AnalogHistory
WHERE TagName = 'SysTimeHour'
AND DateTime = '20070218 17:45:55'
and
SELECT Value
FROM InSQL.runtime.dbo.AnalogHistory
WHERE TagName = 'SysTimeHour'
AND DateTime = '20070218 17:45:58'
both request data 3 seconds apart, and both return a value of 17, even though the tag
‘SysTimeHour’ is stored cyclically only once every 5 minutes.

Wonderware System Platform Course - Part 2

3-10 Module 3 – History Data Storage and Retrieval

Wide Table Transformation

Wide Table Transformation is defined as the ability to provide the same historical data in two
different table formats: one in Narrow table format, and one in Wide table format, for each of the
basic data types.
For example, we can view the tables “WideHistory” and “History”. These two formats are
appropriate for very different uses.
Data for three tags from the wide table typically looks like this:

DateTime SysTimeSec SysTimeMin SysTimeHour

------- ----- -----------------------------------------------------
2007-02-18 13:56:13.153 13.0 56.0 13.0
2007-02-18 13:56:49.517 4 56.0 13.0
2007-02-18 13:57:25.880 25.0 57.0 13.0
2007-02-18 13:58:02.243 2.0 58.0 13.0
2007-02-18 13:58:38.607 38.0 58.0 13.0
2007-02-18 13:59:14.970 14.0 59.0 13.0
2007-02-18 13:59:51.333 51.0 59.0 13.0

The same data from the narrow table looks like this:

DateTime TagName Value Quality

------- ----------------------------------------------------- -------
2007-02-18 13:52:04.277 SysTimeSec 4.0 0
2007-02-18 13:52:04.277 SysTimeMin 52.0 0
2007-02-18 13:52:04.277 SysTimeHour 13.0 0
2007-02-18 13:52:40.640 SysTimeSec 40.0 0
2007-02-18 13:52:40.640 SysTimeMin 52.0 0
2007-02-18 13:52:40.640 SysTimeHour 13.0 0
2007-02-18 13:53:17.003 SysTimeSec 17.0 0
2007-02-18 13:53:17.003 SysTimeMin 53.0 0
2007-02-18 13:53:17.003 SysTimeHour 13.0 0
2007-02-18 13:53:53.367 SysTimeSec 53.0 0
2007-02-18 13:53:53.367 SysTimeMin 53.0 0
2007-02-18 13:53:53.367 SysTimeHour 13.0 0
2007-02-18 13:54:29.730 SysTimeSec 2.0 0
2007-02-18 13:54:29.730 SysTimeMin 54.0 0
2007-02-18 13:54:29.730 SysTimeHour 13.0 0

The data is physically stored only once.

Wonderware Training