Archer Suite: Data Feed Monitor Guide
Archer Suite: Data Feed Monitor Guide
Archer Suite: Data Feed Monitor Guide
Archer® Suite
6.7
Contact Information
RSA Link at https://community.rsa.com contains a knowledgebase that answers common questions and provides
solutions to known problems, product documentation, community discussions, and case management.
Trademarks
RSA Conference Logo, RSA, and other trademarks, are trademarks of RSA Security LLC or its affiliates ("RSA"). For a
list of RSA trademarks, go to https://www.rsa.com/en-us/company/rsa-trademarks. Other trademarks are
trademarks of their respective owners.
License Agreement
This software and the associated documentation are proprietary and confidential to RSA Security LLC or its affiliates
are furnished under license, and may be used and copied only in accordance with the terms of such license and with
the inclusion of the copyright notice below. This software and the documentation, and any copies thereof, may not
be provided or otherwise made available to any other person.
No title to or ownership of the software or documentation or any intellectual property rights thereto is hereby
transferred. Any unauthorized use or reproduction of this software and the documentation may be subject to civil
and/or criminal liability.
This software is subject to change without notice and should not be construed as a commitment by RSA.
Third-Party Licenses
This product may include software developed by parties other than RSA. The text of the license agreements
applicable to third-party software in this product may be viewed on the product documentation page on RSA Link.
By using this product, a user of this product agrees to be fully bound by terms of the license agreements.
Distribution
Use, copying, and distribution of any RSA Security LLC or its affiliates ("RSA") software described in this publication
requires an applicable software license.
RSA believes the information in this publication is accurate as of its publication date. The information is subject to
change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS IS." RSA MAKES NO REPRESENTATIONS OR
WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY
DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. CUSTOMER IS
SOLELY RESPONSIBLE FOR ENSURING THAT THE INSTALLATION OF THE APPLICATION IS PERFORMED IN A SECURE
MANNER. RSA RECOMMENDS CUSTOMERS PERFORM A FULL SECURITY EVALUATION PRIOR TO IMPLEMENTATION.
©
2021 RSA Security LLC or its affiliates. All Rights Reserved.
March 2021
2
Archer Data Feed Monitor Guide
Table of Contents
Release Notes ............................................................................................................................................... 4
Chapter 1: Overview of Archer Data Feed Monitor ...................................................................................... 5
About Archer Data Feed Monitor ............................................................................................................. 5
Key Features and Benefits ........................................................................................................................ 5
Data Feed Monitor Tool Prerequisites ...................................................................................................... 6
Chapter 2: Archer Data Feed Monitor Components ..................................................................................... 6
Architecture Diagram ................................................................................................................................ 6
Chapter 3: Installing and Configuring Archer Data Feed Monitor ................................................................ 7
Installation Overview ................................................................................................................................ 7
Installation and Configuration .................................................................................................................. 7
Test the Connection ............................................................................................................................ 15
Chapter 4: Troubleshooting ........................................................................................................................ 15
Configuration File (DataFeedMonitor.exe.config) .................................................................................. 15
Appendix A: Certification Environment ...................................................................................................... 16
3
Archer Data Feed Monitor Guide
Release Notes
Document Version Published Date Notes
1.0 August 2018 Initial release
1.1 May 2019 The new version of RSA Archer Data Feed Monitor uses the
NLog framework to enable logging. There is no change in
the business logic of how the Data Feed Monitor works,
only a change in the logging functionality. The previous
version of this tool did not use a logging framework and
was writing data to one log file. Over time, this file will
become quite large and difficult to work with. NLog
provides a configurable option for logging such as setting
the maximum log file size and options to configure a Rolling
File Appender to create new log files at certain time
intervals.
1.2 October 2019 Added details for the RSA Archer Record Statistical Monitor
1.3 September 2020 The latest version of RSA Archer Data Feed Monitor has the
following updates:
1. Exceptions that occur while checking the criteria or
executing a feed for a given monitor result in the
rest of the monitors being skipped for that
execution. This bug has been fixed.
2. App Settings in Config file are encrypted when the
utility is run for the first time. Encrypt and decrypt
batch files are no longer required.
1.3.1 March 2021 Updated documentation with the added support for the
use of Stored Procedures in the SQL Monitor.
4
Archer Data Feed Monitor Guide
This use of minutely data feeds produces several undesirable results. Most notably, the rapid growth of
affected system tables: tblAsyncInstrumentation, tblDataFeedHistory, and tblDataFeedMessage. In
some cases, this can result in those tables landing in the top 10 by row count and size on disk.
Spinning up and shutting down a data feed requires system resources. The Job Engine must poll for
runnable jobs, pop the data feed job from the queue, push the next occurrence to the queue, spin up a
thread to process the job, unpack the named argument XML, constitute the data feed, check the source,
determine that there is no work to do, log messages to tblDataFeedHistory and tblDataFeedMessage,
and write the completed job information to tblAsyncInstrumentation. In one client instance, a data feed
that was scheduled to run every five minutes was executed more than 17,000 times during a 60-day
period. During that time there was not a single source row to process. This occupied more than three
days of processing time on the services server and generated thousands of rows of data in the instance
database to track the processing of the feeds. A better solution is needed.
The Data Feed Monitor is intended to be run through the Windows Task Scheduler at a high frequency
(every five minutes or less). When it runs, it checks a pre-configured list of monitors to see if any of
them found work to do. Any monitor that finds pending work will call its associated data feed. This
results in near real-time data feed execution, but without the expense of spinning up a data feed when
there is no work to do. It also avoids the large database footprint associated with frequent data feed
execution.
Revisiting our customer example, the Data Feed Monitor was deployed to watch for pending work. Over
a subsequent 60-day period, the data feed was executed twice, resulting in a major savings of server
resources and database storage.
The Archer Data Feed Monitor is a Tool & Utility that enables an Archer Administrator to call a data feed
only when there is work to do (i.e., records that must be processed), reducing the resource load on the
Archer Platform instance.
Archer Data Feed Monitor is available to both on-premise and hosted customers.
5
Archer Data Feed Monitor Guide
• Implement a monitor that can call the data feed only when there is work to do.
• Reduce the instance processing requirements.
• Reduce the amount of data stored in the database data feed history/log tables.
• Improve instance performance.
Data Feed
Archer to Archer
Filesystem Monitor
Monitor
2) Internal
monitors check 4) REST API queues that data
various record 3) If there is work to do, Data feed for execution. This can
sources to see if Feed Monitor calls a REST include convoys.
there is any work API resource that triggers the
to do. execution of the correct data
feed
6
Archer Data Feed Monitor Guide
It is assumed that the reader has both working knowledge of all products involved, and the ability to
perform the tasks outlined in this section. Administrators must have access to the documentation for all
products in order to install the required components.
Important: The integration described in this guide is being provided as a reference implementation for
evaluation and testing purposes. It may or may not meet the needs and use cases for your organization.
If additional customizations or enhancements are needed, it is recommended that customers contact
RSA Professional Services for assistance.
Logging
The solution makes use of a well-known logging framework called NLog. NLog allows for the creation of
one or more log <target> entries, and one or more associated <logger> entries. The targets define
potential “listeners” for log information, and the loggers define the rules associated with each of them,
including log level. Although a sample config file is provided with the solution, a full discussion of the
configuration file and its use is outside the scope of this document. For more details, please see the
NLog documentation.
.NET Configuration
An ASP.NET console application uses an XML configuration file in the form of the
DataFeedMonitor.exe.config. This file contains the following configuration details:
<appSettings>
<add key="archerBaseUrl" value="https://[YourArcherHostURL]"/>
<add key="cacheFilePath" value="[cacheFilePath]"/>
<add key="cacheFileEncryptionPassphrase" value="[EncryptionPassphrase]"/>
<add key="useSslValidation" value="[true/false]"/>
<add key="securityProtocolVersion" value="[Ssl3/Tls/Tls11/Tls12]" />
<add key="archerInstance" value="[InstanceName]"/>
<add key="archerUser" value="[UserName]"/>
<add key="archerPwd" value="[Password]"/>
<add key="httpTimeoutMilliseconds" value="[httpTimeoutMilliseconds]" />
</appSettings>
7
Archer Data Feed Monitor Guide
<appSettings>
<add key="archerBaseUrl" value="https://localhost/rsaarcher"/>
<add key="cacheFilePath" value="c:\temp\"/>
<add key="cacheFileEncryptionPassphrase" value="d0n7TryTh1s@h0m3!"/>
<add key="securityProtocolVersion" value="Tls" />
<add key="useSslValidation" value="false"/>
<add key="archerInstance" value="Archer"/>
<add key="archerUser" value="adminuser"/>
<add key="archerPwd" value="Password@123"/>
<add key="httpTimeoutMilliseconds" value="30000" />
</appSettings>
Setting Description
archerBaseUrl The root URL for the Archer instance from which information is
pulled.
cacheFilePath Path to the location where the cache file will be written. This flat
file is used to cache Archer ID values. When the solution is
started, values are looked up from the configured Archer instance
and stored in the cache file. This enables correctly referenced
fields, modules, and other resources to use IDs appropriate to the
Archer instance.
cacheFileEncryptionPassphrase If provided, the cache file is encrypted using this setting as the
passphrase, providing security for data at rest.
securityProtocolVersion This value dictates which security protocol should be used.
Options include: SSL3, TLS 1.0, TLS 1.1, and TLS 1.2.
useSslValidation When set to true, custom validation is used for the SSL certificate.
This is necessary when using a self-signed certificate to avoid
validation dialogs.
archerInstance The name of the Archer instance from which information is
pulled.
archerUser The name of the account used to pull information from Archer.
archerPwd The password for the account used to pull information from
Archer.
httpTimeoutMilliseconds The httpTimeoutMilliseconds provides a configuration setting
that enables the user to override the default HTTP timeout of
30,000 milliseconds (30 seconds). This value is in milliseconds and
should be set to the number of seconds you want HTTP time out
to take * 1000.
8
Archer Data Feed Monitor Guide
The configuration file also allows you to define monitors, along with other necessary details.
name – the name of the monitor is used only to allow the administrator to determine what has
been configured. It is recommended that monitors be named for the feeds they will run.
associatedFeedGuid – the GUID associated with the feed that should be run when the
monitor finds pending work.
isFeedConvoy – a Boolean (true/false) value indicating whether or not this feed has others
that reference it. When this value is set to “true”, the next feed will be automatically executed
upon completion of the reference feed.
9
Archer Data Feed Monitor Guide
10
Archer Data Feed Monitor Guide
Procedure:
11
Archer Data Feed Monitor Guide
<monitorSection>
<monitors>
<monitor
monitorType="ArcherRecordMonitor"
name="[Name of the datafeed. Value should be Unique]"
associatedFeedGuid="[Data Feed Guid]"
isFeedConvoy="[true/false]"
archerBaseUrl="http://[YourArcherHostURL]"
cacheFilePath="[cacheFilePath]"
cacheFileEncryptionPassphrase="[EncryptionPassphrase]"
useSslValidation="[true/false]"
archerInstance="[InstanceName]"
archerUser="[UserName]"
12
Archer Data Feed Monitor Guide
archerPassword="[Password]"
reportGuid="[Archer Report Guid]"/>
<monitor
monitorType="ArcherRecordStatisticalMonitor"
name="[Name of the datafeed. Value should be Unique]"
associatedFeedGuid="[Data Feed Guid]"
isFeedConvoy="[true/false]"
archerBaseUrl="http://[YourArcherHostURL]"
cacheFilePath="[cacheFilePath]"
cacheFileEncryptionPassphrase="[EncryptionPassphrase]"
useSslValidation="[true/false]"
archerInstance="[InstanceName]"
archerUser="[UserName]"
archerPassword="[Password]"
reportGuid="[Archer Statistical Report Guid]"/>
<monitor
monitorType="FileSystemMonitor"
name="[Name of the datafeed. Value should be Unique]"
associatedFeedGuid="[Data Feed Guid]"
isFeedConvoy="[true/false]"
filePatternToMatch="[File Type to watch]"
directoryToWatch="[Directory Path to be monitored]" />
<monitor
monitorType="SqlDatabaseMonitor"
name="[Name of the datafeed. Value should be Unique]"
associatedFeedGuid="[Data Feed Guid]"
isFeedConvoy="[true/false]"
connectionString="[a connection string to the SQL DB Source]">
<command
isProc="[true/false]"
isScalar="[true/false]"
commandText="[The SQL Query or stored procedure name to be executed]">
<params>
<param name="[the name of the SQL Parameter]" type="[The SQL DB Type
of the SQL Parameter]" value="[the value of the SQL Parameter]"/>
</params>
</command>
</monitor>
<monitor
monitorType="FtpMonitor"
name="[Name of the datafeed. Value should be Unique]"
associatedFeedGuid="[Data Feed Guid]"
isFeedConvoy="[true/false]"
filePatternToMatch="[Regex identifying the files to be watched]"
ftpServerUrl="[FTP Url]"
isAnonymous="[Data Feed Guid]"
user="[FTP UserName]"
password="[FTP Password]" />
</monitors>
</monitorSection>
Option: Users can also use a separate config file for the <monitorSection>. To use a separate file
for monitor section:
• Edit and replace the <monitorSection> in the xml configuration file with
13
Archer Data Feed Monitor Guide
5. Open NLog.config. Locate file target and provide the following information.
filename= "[logFilePath]"
Example: fileName="c:\temp\DataFeedMonitor_Nlog.log"
Name of the file to write to. Provide name of the file to write to with the full path.
archiveFileName="[archivelogFilePath]"
Example: archiveFileName="c:\temp\logArchive\DataFeedMonitor_Nlog.{#}.log"
Name of the file to be used for an archive. It contains a special placeholder {#} that will be
replaced with a sequence of numbers depending on the archiving strategy. The number of
hash characters used determines the number of numerical digits to be used for numbering
files.
6. Open Windows Task Scheduler and create a task to run DataFeedMonitor.exe according to the
desired schedule. If you are only scheduling a single task to run the application, you can use the
default configuration file. If you need to schedule multiple tasks to run data feeds on different
schedules, store a separate configuration file for each task and pass the name and path of that
configuration file to the application on the command line in the task. Whether using the default
or a separate named configuration file, create unique Archer user accounts for each task
instance so that no other process can authenticate with the same credentials and invalidate
the session being used by your task.
14
Archer Data Feed Monitor Guide
Section Description
General Provide the DataFeed Name
Triggers Create a new trigger by providing the Frequency to run the tool
d. Once all the properties for windows task schedule is complete, click on ok to start
the schedule.
Note: If Data Feed Monitor is configured to run in Windows Task Scheduler, all associated data
feeds referenced in the config file should have start dates several years in the future in Archer to
prevent unexpected executions.
Security Considerations
• Use HTTPS/TLS for secured connection between the DFM Application and Archer.
• Run the DFM application with the least privileged account (Logon Service permission).
• Run the DFM Utility immediately after configuring to encrypt the monitor and app settings in
the config file.
Risk Considerations
The DataFeed Monitor must be executed immediately after configuring. Failure to do so will result in a
risk exposure, leaving parameters entered in monitorSection and appSettings (includes Archer
credentials) in the config file in plain text.
Chapter 4: Troubleshooting
Configuration File (DataFeedMonitor.exe.config)
• Ensure the values in the configuration file match your Archer environment.
• Do not include default.aspx in the Archer URL.
• Note that the Archer instance name is case sensitive.
15
Archer Data Feed Monitor Guide
• If you are changing the Archer URL in the config file, ensure that you have deleted the previously
used cache file.
• Ensure that dedicated, active Archer user account credentials are used in the configuration file.
Others:
• When configuring a file transport monitor, if the directory being monitored is not in the same
location where the Data Feed Monitor is running, ensure that you have specified the full path in
the “directoryToWatch” parameter. This will be the case when the directory to watch is on a
network share, or when the it is not in the same location where the Data Feed Monitor is
running. If the folder has security, you can schedule the task in Windows Task Scheduler to run
under a privileged account. Or you can just have it run the task under a service account and then
grant that account access to the share.
• If you are running an Archer Record Monitor and the utility is reporting that the criteria are not
met, even when you think they should be, ensure that the report that the monitor is watching is
a standard report, not a statistics report. If the report is a statistics report, you must configure
an Archer Record Statistical Monitor instead. This is because the two reports return different
XML schemas, so a specialized monitor is required to correctly parse the XML and count results.
16