Configure SFTP Shell Script

Copyright © 2017 EMC Corporation. All Rights Reserved.

Trademarks
RSA, the RSA Logo and EMC are either registered trademarks or trademarks of EMC Corporation in the
United States and/or other countries. All other trademarks used herein are the property of their respective
owners. For a list of EMC trademarks, go to www.emc.com/legal/emc-corporation-trademarks.htm.

License Agreement
This software and the associated documentation are proprietary and confidential to EMC, are furnished
under license, and may be used and copied only in accordance with the terms of such license and with the
inclusion of the copyright notice below. This software and the documentation, and any copies thereof, may
not be provided or otherwise made available to any other person.
No title to or ownership of the software or documentation or any intellectual property rights thereto is
hereby transferred. Any unauthorized use or reproduction of this software and the documentation may be
subject to civil and/or criminal liability. This software is subject to change without notice and should not be
construed as a commitment by EMC.

Third-Party Licenses
This product may include software developed by parties other than RSA.

Note on Encryption Technologies

This product may contain encryption technology. Many countries prohibit or restrict the use, import, or
export of encryption technologies, and current use, import, and export regulations should be followed when
using, importing or exporting this product.

Distribution
Use, copying, and distribution of any EMC software described in this publication requires an applicable
software license. EMC believes the information in this publication is accurate as of its publication date.
The information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS IS." EMC CORPORATION
MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE
INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Contents

Configure SFTP Shell Script File Transfer 4

Enhancements for Version 3 5

Upgrade the Agent 6

Move Configuration Information 6
Download the Script 6
Move Persistent Information 7
Run the Move Script 7
Confirm the Files Were Moved 7

Install and Configure the Agent 9

Download the Agent 9
Create or Configure a User Account to Run the Agent 9
Create and Update the Configuration File 10

Shell Script Parameters 11

Configuration Script Information 13

Configure RSA NetWitness Log Collector to Receive Log Files 14

Configure SFTP Shell Script File Transfer
Use the sasftpagent.sh shell script to transfer text-based log data from Linux systems. This script takes
data slices from active log files, but only transfers the new data each time the script runs.
Schedule the script in cron to run as often as you want log data sent to the RSA NetWitness Log Collector.
The script uses the SFTP or SCP protocol to transfer the data.
Note the following:
l All connections are initiated from the system to RSA NetWitness Suite.

l The script runs on all POSIX compliant Unix/Linux systems and shells.

Note: You must use OpenSSH version 4.4p1 or later.

l RSA recommends that you set up a cron job to run the script at specified time intervals. However, if you
do set up a cron job, make sure to run it as a user that has access to the logs that need to be sent to RSA
NetWitness Suite.

This topic contains the following information:

l Enhancements for version 3 of the Agent

l Instructions for Upgrading the Agent: follow these steps if you are currently running version 2.7 of the
agent

l Instructions for Installing the Agent: follow these steps if you are downloading the agent for the first
time

l Details for the Shell Script Parameters

l Instructions to Configure the RSA NetWitness Log Collector to Receive Log Files

You must perform the following steps to complete the installation and configuration of the agent:
I. Install or Upgrade the agent, depending on whether or not you are running it currently.

II. Configure the the RSA NetWitness Log Collector to Receive Log Files.
 Configure SFTP Shell Script

Enhancements for Version 3

l The script runs on all POSIX compliant Unix/Linux systems and shells.

l Expects configuration at /etc/rsa/sasftpagent.conf.

l Encourages the user to keep configuration separate from script source. Warning is logged if the user does
not do so.

l Persistent state is written to /var/lib/rsa by default.

l Enhanced user interaction, such as:

l Support for running the script without root privileges.

l The agent has been made highly configurable so that it can be configured to run from anywhere, and
create a persistent state directory anywhere. For example, non-root users can persist state information
to their home directory, by specifying an alternate persistent state directory in the configuration file.

l A configuration specified at a non-root user's home directory (e.g. ~/sasftpgent.conf) is

automatically picked up while running as a non-root user.

l Command line options (-C or --config) have been added to point the agent to custom configuration.

l Logs are written to /var/log/rsa/sasftpagent.log. Logging levels have been introduced so that the
logs may be filtered for WARN, ERROR and FATAL entries, to help troubleshoot issues. These log
entries can now be used to perform troubleshooting after the fact.

l If the user forgets to edit the configuration, a FATAL log entry is generated. The entry contains a clear
message that the user needs to edit the configuration before the script can be used.

Enhancements for Version 3 5

 Configure SFTP Shell Script

Upgrade the Agent

If you have used a version of sasftpagent.sh prior to version 3, then you should follow the instructions in this
section to upgrade to the latest version.
The major steps are as follows:
I. Download the new Agent from RSA Link.

II. Move configuration information to /etc/rsa/sasftpagent.conf.

III. Download the mvpersinfo.sh script.

IV. Run the mvpersinfo.sh script to move persistent information to the location used by version 3.

V. Run version 3 of the agent.

Move Configuration Information

In version 2.7, the user configuration was specified in one of the following two locations:
l The configuration may have been edited inline within /usr/local/sa/sasftpapgent.sh (or wherever
you placed the script)

l The configuration may have been specified separately; wherever the CONFIG_FILE parameter in the
sasftpagent.sh script is set.

For any parameters that you edited within the script or that you specified in the separate configuration file,
you need to move them to a separate file in the following location: /etc/rsa/sasftpagent.conf.
The following parameters are the ones that users often change during configuration:
l SA

l DATA_DIRECTORY

l FILESPEC

l FLAG_REMOVE_AFTER_SEND (only set if you wanted to remove data files automatically after
transferring them to the Log Collector).

Download the Script

1. Navigate to https://community.rsa.com/docs/DOC-45018.

Note: You need to log on with credentials supplied to you by RSA.

6 Upgrade the Agent

 Configure SFTP Shell Script

2. Click mvpersinfo.sh to download the script.

Move Persistent Information

In version 2.7, the persistent state information is maintained in the /usr/local/sa directory by default. It
can also be specified in the PERSINFO_DIRECTORY parameter.
The persistent information directory contains tracking files that contain the number of lines, for each data
file, that have already been transferred to the Log Collector. The agent uses this information to figure out
which lines in each file are new, since the last time it ran. It then transfers only the new data, and updates
the tracking files accordingly.
It is important to move these files into the new location before you run version 3.0.1 of the agent. RSA
provides a script, mvpersinfo.sh, for moving the persistent information.

Run the Move Script

To move the persistent information to its new location, perform the following steps:
1. Copy mvpersinfo.sh to the system where you run version 2.7 of the agent.

2. Open mvpersinfo.sh with a text editor, and confirm that OLD_PERSINFO_DIRECTORY is set to the
value set for PERSINFO_DIRECTORY in your 2.7 configuration.

3. Run the script using the following command:

sh mvpersinfo.sh

If there are no errors, and the script runs successfully, it does not generate any output.

Confirm the Files Were Moved

After you run the script, you can confirm the successful movement by using the following procedure:
1. Run the following command to get a list of the old tracking files:
find /usr/local/sa -name "*-*last.line"

2. Run the following command to get a list of the new tracking files:
find /var/lib/rsa/sasftpagent -name "*-*last.line"

3. Compare the output from the two commands. The output should be similar, with the only difference being
the paths to the files. Here is a sample of the results of running these commands after moving the
persistent files:
$ find /usr/local/sa -name "*-*last.line"

/usr/local/sa/opt/log/bar.log-sa.last.line
/usr/local/sa/opt/log/foo.log-sa.last.line
usr/local/sa/var/log/foo.log-sa.last.line
/usr/local/sa/var/log/fob.log-sa.last.line

Upgrade the Agent 7

 Configure SFTP Shell Script

$ find /var/lib/rsa/sasftpagent -name "*-*last.line"

/var/lib/rsa/sasftpagent/track/opt/log/bar.test-last.line
/var/lib/rsa/sasftpagent/track/opt/log/foo.test-last.line
/var/lib/rsa/sasftpagent/track/var/log/foo.test-last.line
/var/lib/rsa/sasftpagent/track/var/log/fob.test-last.line

8 Upgrade the Agent

 Configure SFTP Shell Script

Install and Configure the Agent

If you are not upgrading from a previous version of the agent, follow the steps in this section to download,
install, and configure the agent.
I. Download the Agent

II. Create or Configure a User Account to Run the Agent

III. Create and Update the Configuration File

IV. Schedule the Agent to Run Periodically: Configure cron or your OS scheduler to automate running the
script at your desired interval.

Download the Agent

Follow these steps to download the SA SFTP Agent (sasftpagent.sh) from RSA Link.
1. Navigate directly to the RSA NetWitness SFTP Agent Downloads URL on RSA Link:
https://community.rsa.com/docs/DOC-45018

Note: You need to log on with credentials supplied to you by RSA.

2. Click RSA NetWitness Unix SFTP Agent and save the file anywhere on your file system.

3. Set execute permissions on the sasftpagent.sh file. For example, run the following command:
chmod 755 /usr/local/sa/sasftpagent.sh

Create or Configure a User Account to Run the Agent

The user account that runs the agent needs read permission to the DATA_DIRECTORY that holds the logs,
and read/write access to the PERSINFO_DIRECTORY that the agent uses for persistent storage.

To create the user account to run the agent:

1. On the logcollector host, run the lc_upload_support script.
l This script is located in /opt/netwitness/bin

l This script creates the sftp user for transferring the log files.

2.

Install and Configure the Agent 9

 Configure SFTP Shell Script

Create and Update the Configuration File

Create the configuration file here: /etc/rsa/sasftpagent.conf. If you do not have a configuration file,
copy the script file (sasftpagent.sh) and remove everything but the configuration parameters. Update the
configuration file with the information for your environment. For reference, see the Parameters table below.
If you are running the script for the first time, run the following command, where collector-IP is the IP
address of your RSA NetWitness Log collector:
sftp collector-IP

Caution: It is important to run this command as the same user who will run it when it is automated.

10 Install and Configure the Agent

 Configure SFTP Shell Script

Shell Script Parameters

The following table describes the most important parameters that you need to set when configuring the script.

Parameter Values Description

SA Name or IP address The name or IP address of your RSA NetWitness Log Collector
host.

DATA_ Directory path or paths, The local source for the log data. For example:
DATA_DIRECTORY=/var/log:/var/log/audit
DIRECTORY separated by colons (:).
You can specify one or more folders.

Note: All folders that you specify are searched for the file
names that you specify in the FILESPEC parameter.

FILESPEC File name or names, File mask that matches the log files to be processed by the
script.
separated by colons (:).
Note: The script supports line-by-line text data. Thus, .xml,
.zip, .gz, .exe and other non-text formats are not supported.

For example, to process all files in the folder:

FILESPEC=isi_webui.log:smb.log:/etc/logs/*.*

Note: The files that you specify can reside in different

folders. Make sure to list all of the necessary folders in the
DATA_DIRECTORY parameter.

SA_ The directory name of The destination folder name. For example:
/upload/apache/muditapache
DIRECTORY your NetWitness Log
Collector host

TRANSFER_ SFTP, or SCP SFTP is the default (and recommended) transfer protocol
METHOD

USEHEAD A non-negative integer, The number of lines in each log file to be considered as a
header that must be transferred to Log Collector in each
representing the number
transfer.
of header lines
You can set this to 0 to indicate that there are no header lines.

Shell Script Parameters 11

 Configure SFTP Shell Script

Parameter Values Description

DEPTH A positive integer, rep- Governs how many levels deep the script searches to find logs
under the configured DATA_DIRECTORY.
resenting number of
Defaults to DEPTH=1, which causes the script to search for
folder levels
data files directly under the directories configured in DATA_
DIRECTORY, but not in any sub-folders.

SFTP and SCP Settings

USERNAME sftp Default setting for SSH daemon on the RSA NetWitness Suite
platform.

IDENTITY File path Location of the private key used to connect to RSA NetWitness
Suite. For instructions on generating keys, see Install and
Update the SFTP Agent.
The default value is the following:
$HOME/.ssh/id_rsa

12 Shell Script Parameters

 Configure SFTP Shell Script

Configuration Script Information

All configuration settings can be loaded using a configuration file that is separate from the script. This file
should contain one setting and one value per line (except for DATA_DIRECTORY and FILESPEC, which
can contain multiple colon-separated entries).
The configuration file must be located in the directory assigned by SA_DIRECTORY in the shell script or in
the path of the shell that calls the script. The SA_DIRECTORY can be overridden in the configuration file,
although the shell script will try to use its own SA_DIRECTORY setting to open the configuration file.
For example, a configuration file could contain the following information:
SA=10.31.246.168
DATA_DIRECTORY=/var/log/httpd
SA_DIRECTORY=/upload/apache/muditapache
USERNAME=sftp
FILESPEC=*_log*
FLAG_REMOVE_FILE_AFTER_SEND=no

Configuration Script Information 13

 Configure SFTP Shell Script

Configure RSA NetWitness Log Collector to Receive Log

Files
Before you configure the Log Collector, remember to run the following command on any new event source,
from which RSA NetWitness Suite has not previously collected logs using SFTP or SCP:
sftp collector-IP

where collector-IP is the IP address of your RSA NetWitness Log collector.

To configure RSA NetWitness Suite to receive the log files:

1. On the Linux or Unix event source, run the following command to generate the public/private key pair:
ssh-keygen -b 1024 -t rsa

This command creates id_rsa in OpenSSH format, which is used by RSA NetWitness Suite. If your Linux
system creates IETF SECSH format by default, run the following command to convert it:
ssh-keygen -f ~/.ssh/id_rsa.pub -i

2. Add the public key into the log collector, as described in Install and Update the SFTP Agent.

14 Configure RSA NetWitness Log Collector to Receive Log Files