Thales - Data Transformation
Thales - Data Transformation
- Data Transformation
1. Document Information
Document may be freely distributed within Thales E-security and its subsidiaries.
1.4. Approvals
1.5. Distribution
The following review matrix shows who this document has been distributed to and what
sections those individuals are reviewing:
2. Table of Contents
3. Introduction .......................................................................................................... 5
3. Introduction
This document details the Vormetric Data Security software suite and will as an operational
(run book) guide for managing the Vormetric security environment. This guide will contain the
steps to encrypt data and reverse data encryption by our File System (FS) agent. Using the
various methods including Copy Method, Offline DataXform tool and Live Data
Transformation (LDT) feature. LDT is a Vormetric Transparent Encryption (VTE) agent feature
that allows the automated transformation of data without the need to stop the application or
users accessing the data. The key features are to minimise downtime for applications and
users while the data is being transformed. In addition, LDT provides tools to schedule the
rotation of encryption keys for enhanced security, should the key become compromised.
1. Copy Method, this requires an empty guard path and a standard encryption policy
that facilitates the encryption in the copying process.
2. Offline Data transformation utilises a special data transformation policy and the
dataxform tool.
3. Live Data Transformation, is a new license feature that allows the transformation of
files and folders while users and applications are accessing the guard path.
In this section are the steps to encrypt data using the Copy method. The process is to copy
data from one location that is not under Vormetric control to an empty folder that is
guarded with a Vormetric Wide-Open policy. The policy is wide-open and allows any user
and process to encrypt the data during the copy process. The diagram below illustrates at a
high level on how the process works.
The copy tool (i.e. Windows Explorer or Robotcopy) will copy the files from an unguarded
folder to the destination, which is a folder guarded with the Vormetric Encryption Policy. As
the file is read from the file system into memory it remains in the clear. Only when the file is
written back to the file system that our Vormetric Kernel module will apply the encryption
key and encrypt the data.
The best method is to create a Wide-Open security policy to prevent any mistakes or data
corruption in the copy process. Once the data is encrypted the Wide-Open security policy
can be replaced with a more restrictive operational policy.
The advantage of the copy method is that it minimises the downtime for users and
applications and allows the data to be available in parallel. The copy method is also the best
method to transform a directory with a very large number of small files (<1 MB). The
disadvantage is that the data integrity must be maintained by a phased copy process and
additional storage is required.
4. From the drop down select Standard Policy type and enter the name
“wide_open_policy”.
NOTE: Do not create a User Set or Process Set for a Wide_open Policy.
13. Select the encryption key that is to be used for the encryption of the data. Click
Select.
14. Then click OK to add the key to the policy. Click Apply and OK to create the Policy.
15. Next click on the Hosts and select the Host to be encrypted.
17. In the drop-down menu select the “wide-open” policy. Then click Browse.
18. Select the empty folder where the data will be encrypted (i.e. S:\GP).
20. The Guard point will be RED, click Refresh until the Guard Path is GREEN.
21. Once the guard point is active. Simply copy the data from its current location to the
Guard Point. This will encrypt the data.
22. Next once the data has been encrypted. Unguard the Guard Path.
23. Allow the Guard path to disappear. Now the data is unguarded you can verify that it
is encrypted by reading the files with Notepad.
24. Next click Guard and guard with the Operational Policy.
In this section are the steps to decrypt data using the Copy method. The process is to copy
data from a guard point, that is under Vormetric control to an empty folder that is not
guarded using a Wide-Open Policy. The policy is wide-open and allows any user and process
to decrypt the data during the copy process. The diagram below illustrates at a high level on
how the process works.
Essentially the steps are the same as the section above, but the data is being copied from
the Guard Point to a non-guarded location. The Wide-Open policy allows any user or process
to apply the key and revert the encryption so the data is in the clear at the destination.
In this section are the steps to encrypt (transform) data using the offline data transformation
tool (dataxform). This requires the creation of a special offline data transformation policy
and applying the policy to a folder that already contains data in the clear. This allows users
to apply in place encryption to files and folders. An advantage over the copy method is that
the additional storage is not required. However, the disadvantage is that it will require down
time for any users and applications while the data is being transformed. The diagram below
illustrates at a high level on how the process works.
The dataxform tool will transform the files in situ of the guarded folder with the Vormetric
Data transformation Policy. As the file is read from the file system into memory the
dataxform tool applies the clear key so the data remains in the clear. Only when the file is
written back to the file system does the dataxform tool apply the encryption key and encrypt
the data.
The tool handles the files in a batch process up to 10 files at a time. Once the data is
encrypted the data transform policy can be replaced with a more restrictive operational
policy.
4. From the drop down select Standard Policy type and enter the name
“dataxform_policy”.
5. Under the Security Rules section click Add to create the first rule.
10. Click OK to add the rule. Now the Policy will change to an Offline Data
Transformation Policy and the Key Selection Rule will add a new Transformation Key
Box.
11. Next click Add under Security Rules to add a second rule.
19. Select the “Clear Key” and click Select Key. Then click OK.
20. Next under the Data Transformation Rules section click Add.
21. In the Key Add section click “Select” next to Key.
22. Select the encryption key that is to be used for the encryption of the data. Click
Select key.
23. Then click OK to add the key to the policy. Click Apply and OK to create the Policy.
24. Next click on the Hosts and select the Host to be encrypted.
26. In the drop-down menu select the “dataxform” policy. Then click Browse.
27. Select the empty folder where the data will be encrypted (i.e. S:\GP).
29. The Guard point will be RED, click Refresh until the Guard Path is GREEN.
30. Next logon to the Host and launch an administrative command prompt (Windows)
or ssh as Root on to the Linux host.
31. Change director to “C:\Program
Files\Vormetric\DataSecurityExpert\agent\vmd\bin” on Windows and
“/opt/Vormetric/DataSecurityExpert/agent/vmd/bin” on Linux.
On Windows:
On Linux:
33. At the prompt type “Y” for yes and hit enter to start the process.
If Shadow Copy was used on your system, you must back up your data before attempting to
run dataxform. Once DataXform has been completed, you may restart Shadow Copy.
Note, however, that all Shadow Copy backups made prior to running dataxform will be
unusable and should be discarded. Attempting to restore your cleartext Shadow
Copy backups made prior to running dataxform into your encrypted data will result in data
corruption.
34. The dataxform tool will provide a summary of the files that have been transformed
and the time taken.
The data transform program ran from Fri Jan 10 12:28:13 2020 until Fri Jan 10 12:28:15 2020
C:\Program Files\Vormetric\DataSecurityExpert\agent\vmd\bin>
35. Now the data has been transformed you will need to unguard the offline
transformation policy. Under the hosts Guard FS tab, select the dataxform policy and
click on unguard.
IMPORTANT NOTE: Remember to use the same encryption key from the Offline Data
Transformation Policy in the Operational Policy.
In this section are the steps to use the Offline Data Transformation tool (dataxform) to
remove the encryption from files and folders under a guard point. The steps are like those in
section 5 for Offline data Transformation and requires downtime on the data during the
transformation process. You can either create a reverse-dataxform policy or edit an existing
dataxform policy and reverse the encryption keys.
Essentially to remove encryption from the data the dataxform tool will use the encryption
key to read the data into memory in the clear and then write the data back using the clear
key to leave the data in the clear.
NOTE: Make a note of the Encryption key used on the Guard Point, this key will be required
to reverse the encryption.
Here are the steps to edit an existing Offline data transform policy (created in section 5) and
reverse the keys.
4. In the “Clone this policy as” box enter a new policy name like “Dataxform-Reverse”
and click Clone.
8. Select the encryption key that was used for the encryption of the data. Click Select
key.
NOTE: Select the Encryption key that was deployed by the operational policy to encrypt
the data inside the guard path.
9. Now select the Clear key Rule (rule 1) and click delete to remove the Clear key.
10. Next under the Data Transformation Rules section click Add.
11. In the Key Add section click “Select” next to Key.
12. Select the “Clear Key” and click Select Key. Then click OK.
13. Now under Transformation Key Rules, select the Encryption Key Rule (rule 1) and
click delete to remove the Encryption key and leave the Clear Key.
14. The Dataxform_Reverse Policy should now have the keys reversed. The encryption
key for the Key Selection Rule and the Clear Key for the Transformation Rule.
15. Next you will need to stop all users and applications accessing the guard point on
the host. Then unguard the operational policy and re-guard with the
dataxform_reverse policy.
16. After the policy has gone GREEN move to the Host.
17. Next logon to the Host and launch an administrative command prompt (Windows)
or ssh as Root on to the Linux host.
19. Next run the dataxform tool to transform the data. To remove the
dataxform_auto_lock file. This file is created when previous runs of the dataxform
tool and prevents further attempts of the tool being executed.
20. To remove the dataxform_auto_lock file run the dataxform tool with the –cleanup
switch. At the prompt type “y” for yes and hit enter.
21. Once the dataxform status file has been removed you can now run the dataxform
tool to reverse the encryption on the files.
22. Run the dataxform tool with these switches, dataxform.exe --rekey --gp S:\GP --
preserve_modified_time --cleanup_on_success. The switch “—cleanup_on_success”
will automatically remove the datatxform_auto_lock file after a successful run. At
the prompt type “y” for yes and hit enter.
If Shadow Copy was used on your system, you must back up your data before attempting to
run dataxform. Once DataXform has been completed, you may restart Shadow Copy. Note,
however, that all Shadow Copy backups made prior to running dataxform will be unusable
and should be discarded. Attempting to restore your cleartext Shadow Copy backups
made prior to running dataxform into your encrypted data will result in data corruption.
23. The dataxform tool will reverse the encryption on the files, then at the prompt type
“y” for yes and hit enter to remove the dataxform auto lock file.
The data transform program ran from Fri Jan 10 16:08:31 2020 until Fri Jan 10 16:08:33 2020
In this section are the steps to create a Live Data Transformation (LDT) policy and encrypt
data at rest. Like the Offline Data Transformation policy, the LDT policy is a special policy
type that was introduced with Version 6.x of the Vormetric Transparent Encryption (VTE)
agent. This allows users to apply in place encryption to files and folders. An advantage over
the offline transform method is that the applications and users can access the data while the
transformation policy is running. Another key advantage is that the encryption key is
versioned and allows the automatic key rotation. However, there will be an initial downtime
to allow the policy to be applied. Once the policy is status is GREEN, then users and
applications can be given access to the Guard point.
LDT uses a special versioned symmetric key that can be rotated either manually or via a
schedule measured in days. Like the dataxform tool, LDT scans the files and folders inside
the guard point and using a batch process reads the files into memory using the clear key
and transforms the data with the encryption key when writing the files back to the guard
point.
Information concerning the encryption key and its version is stored in the file attributes as
part of the Windows alternate data stream (ADS). This information allows LDT to keep track
of the files encrypted and what version of the key that was used. This information forms part
of the LDT report statistics that are available to system administrators and Vormetric
support.
Here are the steps to create an LDT policy and transform the data.
4. From the drop down select Live Data Transformation Policy type and enter a name
like “LDT_policy”.
5. This will automatically create a Policy with the first Security rule created. This rule
will be any User, any Process, all_ops action with Permit and apply key effect.
6. Edit the Key Selection Rules by adding additional security rules like standard
operational policy with the last rule being the deny rule.
7. The Key Selection Rule will be empty and there will not be any encryption keys
defined.
8. Click Add to define the encryption keys.
9. On the Add keys page click “Select” next to the Current Key.
12. Select the LDT key and click Select Key. Click OK to add the Keys
13. Now the policy is ready, click Apply and OK to create the LDT policy.
14. Next click on the Hosts and select the Host to be encrypted.
16. In the drop-down menu select the “LDT” policy. Then click Browse.
17. Select the data folder where the data will be encrypted (i.e. S:\GP).
19. The Guard point will be RED, click Refresh until the Guard Path is GREEN.
NOTE: The guard point status will initially state “Not Started”, soon once all the files
have been scanned the batch transformation process will start and the % complete will
be shown. Once the rekey on all the files is complete will the status show “Rekeyed”. In
some circumstances if the guard point contains unsupported file or READ ONLY files then
the rekey status will be “Incomplete”.
20. Check the status of the rekey process by clicking on the GREEN Status symbol to
present an LDT report on the guard point.
In this sub-section are the steps to create an LDT key. These are special version keys that are
exclusively for LDT policies.
4. Enter a Name for the LDT key, then click on the calendar next to Expiration Date.
5. Next set the Expiration Date for the initial Version (Version 0) of the LDT key.
NOTE: The expiration date for the initial version 0 of the LDT key is when the automatic
key rotation schedule will start. The expiration date for an LDT key will change and based
on the “Automatic Key Rotation” value in days. So, if the key rotation is set to 365 days
then the LDT key will expire and rotate 365 days from the initial expiration date.
25. Verify that the files can now be seen on the clear.
In this section are the steps that could be taken to reverse the data encryption using an LDT
policy. The potential befits for this process is to use LDT to identify the files that are already
encrypted and not to transform any files that were skipped in the original transformation
process. LDT will not transform any files that are:
In the event that some files have one or more of the above conditions, these files will be
skipped during the LDT transformation. Issues can arise when trying to reverse the
encryption on these files as the traditional dataxform utility may incorrectly transform these
skipped files. A process is required whereby an LDT policy can be used to reverse the
encryption on files that were originally transformed by LDT.
In the example below a Guard Point was transformed by an LDT policy and there were
several files skipped, this leads to an LDT transformation status of “incomplete”. The
traditional process would be to copy these skipped files out of the guard point before
applying a dataxform policy and running the dataxform utility to transform the remaining
file. This process is effective but has one major flaw, scalability. If the guard point contained
around 1 TB of unstructured data, the task of moving skipped files would be a major
undertaking.
The advantage of using an LDT policy is that those skipped files will be skipped again when
data transformation is executed.
In this section are the steps to create an LDT decryption policy. LDT policies use a versioned
encryption key, this facilitates the automation of the key rotation process. LDT versioned
encryption keys can only be used in LDT policies in the transformation key selection. In order
to create a non-versioned LDT key, for a decryption policy, it must first be cloned.
1. Check the original policy and notate the key used to encrypt the data.
2. Navigate to Agent Keys/Keys and select the LDT key used. In the Versions tab, select
the latest version and click Clone.
3. The new key name should end with the version number that matches the version as
follows. Click OK to clone. An error will occur if you forget to enter a Key Refresh
Period between 1 and 44640 (seconds). The default is 10080.
4. Next create a new LDT policy and give it a name like “LDT_Decrypt”. Navigate to
Policies > Manage Policies. Click Add. From the Policy Type drop down select Live
Data Transformation. This generates an LDT policy with the first security rule already
populated with the key ops action.
5. Create a second security rule with action “all_ops” and Permit, Apply Key Effect.
6. Under the Key selection rule click Add. For the Current key, click Select and select
the cloned LDT key from step 2 above. Click Select Key.
7. Under the Transformation Key click select and select the clear key. Click Select key
and OK. The policy is now complete. The original encryption key will be swapped out
for the clear key.
In this section are the steps to decrypt the data using the LDT decryption policy created in
the previous section. This process is suitable for situations were some files were skipped
during the initial decryption. The reason for the files being skipped should be that they were:
NOTE: Do not follow these steps if the files were skipped because they were open. These
open files will be transformed when the rekey process runs again.
1. First unguard the active LDT policy. Navigate to the Host > Guard FS and select the
active Guard Point and click unguard. Please ensure that all applications and users
are no accessing the guard point. In the example shown the LDT rekey status is
incomplete. This was due to some file being skipped because they were not
supported.
2. Wait for the Guard point to be deleted and disappear. This may require a reboot of
the client.
3. Now that the Guard Point is removed it is important that all applications and users
are stopped from accessing the data.
4. Next verify the LDT metadata file is present inside the Guard Point directory by
running this command.
On both Windows and Linux, run “voradmin ldt attr delete <GP>”
6. Now with the LDT metadata deleted create a Guard Point using the LDT decryption
policy on the same directory.
8. The LDT rekey process will start, allow the transformation to complete. In those
cases, where the original rekey was incomplete due to unsupported files. The
decryption rekey will also be incomplete. The skipped files will remain unchanged.
9. Once the rekey has completed and transformed all the encrypted files. The next step
is to unguard the decryption policy. Navigate to the Host > Guard FS and select the
active Guard Point and click unguard.
10. Now that the guard point has been removed verify that the encrypted files have
been transformed to clear.
11. Now remove the LDT metadata by running this command:
On both Windows and Linux, run “voradmin ldt attr delete <GP>”