Self-Service Supportability Orchestrator Product Guide
Self-Service Supportability Orchestrator Product Guide
Self-Service Supportability Orchestrator Product Guide
COPYRIGHT
Copyright © 2023 Musarubra US LLC.
TRADEMARK ATTRIBUTIONS
Trellix and FireEye are the trademarks or registered trademarks of Musarubra US LLC, FireEye Security Holdings US LLC, and their affiliates in the US
and /or other countries. McAfee is the trademark or registered trademark of McAfee, LLC or its subsidiaries in the US and /or other countries. Other
names and brands are the property of these companies or may be claimed as the property of others.
Contents
Contents ........................................................................................................................................................ 1
Self-Service Supportability Orchestrator (SSSO) ....................................................................................... 2
Supported operating systems ................................................................................................................... 3
Folder Structure ........................................................................................................................................ 4
SSSO GUI Workflows ................................................................................................................................. 5
Playbook Execution Workflow (Running the SSSO Playbooks on a Host) ............................................ 5
Collect MER Workflow ........................................................................................................................ 14
Create EEDK Workflow........................................................................................................................ 17
Update Tools Workflow ...................................................................................................................... 19
SSSO Execution History ........................................................................................................................... 23
Settings.................................................................................................................................................... 24
Proxy Server Settings .......................................................................................................................... 24
SSSO Telemetry ................................................................................................................................... 25
Command-Line Parameters .................................................................................................................... 25
Running SSSO in Air-gapped Environments ............................................................................................ 27
How do you execute a playbook and record your observations? .......................................................... 27
ePO Endpoint Deployment Kit (EEDK) .................................................................................................... 29
How to create an EEDK package for ePO deployment?...................................................................... 29
How to Deploy SSSO to multiple hosts from ePO? ............................................................................. 29
SSSO usage on MACC enabled systems .................................................................................................. 30
Common Error Scenarios ........................................................................................................................ 30
Another Instance of SSSO Already running......................................................................................... 30
Unsupported OS failure ...................................................................................................................... 30
Powershell version is less than 3.0 ..................................................................................................... 31
Validation Failed ................................................................................................................................. 31
Case Sensitivity Validation Failed........................................................................................................ 31
Playbook Execution failed ................................................................................................................... 32
Playbooks for Data Collection ................................................................................................................. 33
Default Playbooks ............................................................................................................................... 33
Author your own playbook ................................................................................................................. 44
1
Self-Service Supportability Orchestrator (SSSO)
The ultimate key to solving any escalation is DATA. But what happens when you give someone a difficult
time to collect DATA? What if there is an issue that requires monitoring the processes constantly for
inputs?
The Self-Service Supportability Orchestrator (SSSO) tool helps us keep a check in such scenarios. It is a
data collection tool, that brings several supportability tools used by endpoint customers under a single
orchestrator thereby invoking the right tool at the right time, thereby reducing the data collection
effort.
For example:
o When you want to capture system data using both Trellix and Microsoft tools together. Such a
scenario arises when the system CPU utilization is high or when a third-party application fails to
open a browser and crashes due to access denial.
o When you enable debug logging for Trellix® Endpoint Security and Trellix® Agent to collect the
process dump of a certain third-party process which fails within minutes of being initiated. It is
also used to collect files when a process fails. SSSO stops logging information until the process
starts again. Also, you want to collect files only if the process crashes otherwise stop logging
until the process starts again.
2
Supported operating systems
Operating system Versions Compatibility
Windows 11 Yes
Important: SSSO can only run-on above OS versions. Running SSSO on older OS versions will throw an
unsupported OS error.
Note: Before you begin, please make sure you have Windows PowerShell version 3.0 or above
installed. In this release, SSSO supports Collect MER Workflow (Please refer Collect MER Workflow).
If you want to use SSSO only for this workflow, PowerShell version upgrade is optional.
PowerShell is shipped with the applicable operating system by default. Verify the version of PowerShell
and upgrade it to 3.0 or later. Also, If Exploit Prevention is enabled, then Execution Policy Bypass in
PowerShell rule should not be configured to Block.
3
Folder Structure
Folder Structure File and Description
doc: The doc folder consists of the help documents. Playbook.yml
contains the steps to create your own playbook pertinent to the
issue. TelemetryData.txt contains details about telemetry
information collected for product improvement and technical
support purpose. This file gets created in the root folder.
playbooks: Playbooks consists of two subfolders: custom and
default. In the default folder, you will find a few playbooks which
have been authored based on customer escalations. The custom
folder is to be used when a new playbook has been authored.
plugins: Plug-ins consist of two subfolders: custom and default. In
the default folder, there are a few PowerShell scripts which monitor
specific use cases. The plug-ins are basically PowerShell modules.
You can always write your own plug-in and save them to the custom
folder. There is a plethora of resources on the internet for
PowerShell scripting. Note: The default plugins will be checked for
signing while execution, whereas the custom plugins are not
checked for signing. Trellix will not be held responsible for use of
custom plugins by a customer.
run: By default, the run folder is not present when the tool is freshly
extracted from the SSSO_1.0.0.xxx.zip. The folder gets created
automatically when SSSOLauncher.exe is executed. This folder
contains the output of the command executed in a compressed .tgz
format saved as per the timestamp. This compressed file holds the
SSSO.log, SSSO_Telemetry.xml and other dump files or logs
depending on the tools used in the command execution.
tools: The tools folder is categorized into custom and default, which
have subfolders within them namely x64 and x86. The default tools
are shipped with all relevant Trellix tools. For the first time, the
custom tools are automatically downloaded when you run a
playbook. The tools can be updated to latest versions by running
the Update Tools Workflow or by running SSSOLauncher.exe
-update command. To use third party tools, user needs to accept
the third-party license agreements when prompted. You can also
place other needed tools into the custom folder. Note: The default
tools will be checked for signing while execution, whereas the
custom tools are not checked for signing. Trellix will not be held
responsible for use of custom tools by a customer.
License.txt: License information of components used. Added as a
reference.
SSSOLauncher.exe: This is the main executable which is used to
launch the SSSO application. It also accepts command-line
4
arguments. See the help.txt document in the doc folder to know
more about the input arguments for the binary.
SSSO.log: This is the log file generated after running
SSSOLauncher.exe. It displays the log output of the last command
execution only.
SSSO_Updater.log: This is the updater log file generated after
running SSSOLauncher.exe. It displays the auto-update and log
upload output of the last executed command only.
SSSO_Telemetry.xml: This xml file displays the telemetry output of
the last command executed.
Royalty-Free Tools License.txt: This is the Self-Service
Supportability Orchestrator end-user license agreement. By
default, this file is not present in the root folder. But when executed
for the first-time, the EULA page is displayed and once accepted,
this file is created in the root folder.
3. Run SSSOLauncher.exe. It launches a splash screen and checks for new SSSO versions.
4. If it encounters an issue while downloading the new package, it displays an auto update error
screen. You can choose to run the current SSSO version.
5
5. The SSSO End User License Agreement (EULA) page is displayed. Once you reach the end of the
license agreement, the Accept button is enabled. Please click Accept to accept the EULA.
6. Once you accept EULA, SSSO navigates to the Playbooks selection page. All default playbooks
are listed under Trellix Default section and if there are any custom playbooks, they are listed
under Custom playbooks section.
6
The description column provides an overview of what each default playbook does, and
the tools used to collect the data.
If there are new custom playbooks added, you can click Refresh Playbook List button
and the new playbooks are loaded in the custom playbook list.
7
All the pages will show free space available on the bottom right corner of the screen.
Note: Please note that SSSO would need minimum of 5 GB free space. SSSO won’t start If the disk
space is less than 5 GB.
The Collect MER workflow supports the execution of MER tool with all its GUI options. Please refer
to Collect MER Workflow for all the details.
In the Trellix Default section, you have 2 other workflows: CreateEEDK and UpdateTools. These are
discussed in detail in Create EEDK Workflow and Update Tools Workflow sections.
Filtering Playbooks
Playbook selection page provides ability to search and filter playbooks. At the top right of the
screen, you have a filter search box. A list of playbooks is displayed based on the keywords
entered in the search box.
For example, when you type “cpu”, all playbooks related to CPU are listed for selection. The
filter uses both playbook name and description as search criteria to find a suitable match.
Once you select a playbook, SSSO navigates to the corresponding parameter page.
Parameter page shows all the relevant parameters required to run the playbook. The
parameters change as per the playbook requirements. For example, a high CPU playbook would
8
require the monitoring process name and the corresponding CPU threshold required to trigger
the data collection.
9
Parameter List and accepted values
10
You can get more information about the parameter, by placing the cursor on the icon
For example, the CPU Threshold should be a value between 1-100 whereas the value entered is 101.
SSSO displays an error highlighting the same.
11
For SR number, SSSO will validate the SR number and highlight any errors detected. Please ensure that
the SR number provided is a valid one. If the SR Number is not valid, SSSO will display the following
error:
Note: If you do not want the data to be uploaded to the SR, please uncheck the Upload to Trellix when
complete checkbox.
Once all the parameters are provided, please click Run Playbook page.
Playbooks require additional third-party tools to collect data. A Download Third-party Tools popup
appears while the tools are being downloaded.
12
The third-party tools get downloaded to the following location: \SSSO_1.0.0.xxx\tools\custom folder.
When you initiate the download of each third-party tool, a window pops up asking you to accept the
end-user license agreement. EULA needs to be accepted when prompted.
If the end-user license agreement is already accepted on the system for a tool (procdump, procmon and
so on), it will not be prompted further. The help dialog would still be displayed.
Note: Trellix has no license to ship third-party tools. As part of SSSO playbook execution, the
third-party tools are downloaded from the internet.
Furthermore, the Playbook progress page displays the summary of SSSO backend playbook task
execution.
13
Data collected by self-service supportability orchestrator is compressed and can be found at
SSSO_1.0.0.xxx\run\<timestamp>.tgz.
The View Logs Folder points to the location of tgz file.
To navigate to the SSSOPlaybook selection page, click Back
Delete the SSSO_1.0.0.xxx folder once the session is complete.
1. SSSO supports WebMER tool workflows in both GUI and command line.
2. In GUI, you can select Collect MER option in playbook selection page.
14
3. The parameter page will show the MER GUI options as below.
SSSO can auto detect the supported Trellix products installed or you can select the specific/all
products from the list.
15
4. You can also select the Event Log Collection duration by clicking Configure button in Event Log
Collection section.
The default values for Application logs are 10 days, System logs is 20 days and Security Logs is 30
days. You can change these values by selecting the appropriate logs and changing the duration.
If you uncheck ‘Use Defaults for Event Log Collection’ and don’t select specific event logs, those
logs won’t be collected. For example, if you selected ‘Collect Application Logs’ and ‘Collect
System Logs’, but didn’t select ‘Collect security Logs’, Application and System logs will be
collected but Security logs won’t be collected.
5. if the ‘Remove IP address, MAC address, Domain names, and Computer names from the MER
result file’ is enabled, SSSO will sanitize the collected data by removing IP addresses, MAC
addresses, Domain names and Computer names from the collected MER data.
6. For Custom Sanitization Settings options, click Configure button in this section.
You can provide a regular expression in Boost Perl syntax: You can either
1. Provide a regular expression (comma separated) or
2. Choose a regular expression file path
NOTE: A user can select to browse to a text file (.txt format) that contains the
regular expressions. Each line in the text file contains only one regular expression.
16
The regular expressions a user must follow are 'boost regular expressions of Perl syntax type'.
For details about Regular Expression Syntax, see the Boost site.
Examples:
action(\w+).dll When a user wants to mask all (.dll) files that start with
action. In this example, \w+ represents a word.
(\w+)\.saleszone\.internalzone\.com When a user wants to mask all server addresses that have
the domain saleszone.internalzone.com.
7. Once you selected the required options, click Start to collect the MER. While the MER collection
is in progress, you can cancel MER execution by clicking the close button of SSSO. You will be
prompted for confirmation.
8. MER Workflow is also supported through command line. Please refer to Command line options
table for the list of all MER command line options.
1. To deploy SSSO on multiple systems, you can create a SSSO EEDK package with the GUI workflow.
2. In the playbook selection page, select CreateEEDK.
17
3. EEDK package is deployed from Trellix ePO on multiple endpoints. As SSSO packages multiple
third-party tools, User should have rights to deploy these third-party tools on the target
machines with System account privileges. If the user has permissions, please read, and accept
the below agreement.
18
4. If third-party tools are not downloaded already, SSSO downloads the third-party tools.
Note: Trellix has no license to ship third-party tools. As part of SSSO EEDK package creation,
the third-party tools are downloaded from the internet.
6. Click View EEDK to access the EEDK package (Stored in the root folder).
This workflow is used to update your third-party tools. If you have an older version and you might want
to update to newer versions of the tools.
Please note that in other workflows, the tools are downloaded if not done already. They won’t be
upgraded automatically. So, if you need updated tools, you can use the Update Tools Workflow to get
the latest versions.
19
In playbook selection page, please select Update Tools workflow.
Next, SSSO displays the list of tools updated. Please read through the agreement and click the accept
button. Once you click the accept button, the third-party tools are updated.
Note: Trellix has no license to ship third-party tools. As part of update tools workflow, the third-
party tools are downloaded from the internet.
20
Once you accept, the tools will be updated. You must accept the tool specific EULAs to update the tools.
Like the playbook workflow, a Download Third-Party Tools popup appears while the tools are being
downloaded.
21
Help Dialog
Help can be accessed by clicking Help button in the right bottom corner as highlighted below.
22
How to use SSSO will take you to the SSSO usage KB92519.
Command-line Help will display all SSSO command-line options to run the tool. For all command-line
options, please refer to Command-Line Parameters section.
23
You can select the required log and click
Settings
Proxy Server Settings
You can configure proxy server settings in SSSO. This might be required for successful uploading of the
data to Trellix SR server and in auto upgrading SSSO to the latest version.
Use system proxy settings – Uses the proxy settings retrieved from the machine.
Configure proxy server – Uses the proxy settings with specified URL and port.
24
Proxy settings once set are stored for future SSSO runs.
SSSO Telemetry
During SSSO execution, it collects execution telemetry details and sends them to Trellix. This helps
Trellix to understand the nature of issues customers are experiencing.
You can turn it off by unchecking the Send telemetry data to Trellix check box.
You can also turn if off by providing skiptelemetry switch in the command line interface.
Command-Line Parameters
Command line: - SSSOLauncher.exe [options.]
Options Description
-? | /? | -help Displays help
-update Download/Update third-party tools
-procname:ProcessName Specify the process name to be used inside the playbook (For
example: -procname:mcshield.exe)
25
-filepath:"<filepath>" Specify the filepath to be used inside the playbook (For example:
-filepath:"C:\test\Logs")
-play playbookname.yml Run the specified playbook (For example: -play CreateEEDK.yml)
-accepteula To silently accept SSSO EULA
-threshold:<value> Specify the CPU threshold to be utilized in the playbook (For
example: -threshold:20)
-skipupgrade Skip auto upgrade of SSSO binaries for a particular run
-sr:<SR Number> Use this switch to upload results automatically to a Trellix Server
against the given valid Service Request number (For example -
sr:4-123456789). You can use it for SSSO playbook or MER
workflow.
-delay:<value> Specify the delay (in seconds) to be used inside the YAML (For
example: -delay:200)
-thresholds:<val1,val2,val3> Specify multiple memory thresholds (in MB) separated by a
comma to be used inside the YAML (For example: -
thresholds:100,200,300)
-createeedk To create an ePO deployable package.
-acceptthirdpartyeula Silently accept Procmon and Procdump EULA when deployed
from ePO (acceptthirdpartyeula switch only works from ePO).
-proxy:<options> Sets the proxy server details. It can be one of below
options.
-proxy:disable
Disables the proxy settings, connects directly to internet
without proxy.
-proxy:system
Applies the system proxy settings.
-proxy:custom(url,port)
Sets the proxy to specified URL and port. (For example: -
proxy:custom(proxy.example.com,9090))
-skiptelemetry Turn off SSSO telemetry.
-MER Runs MER for all detected Trellix products with event logs for
default number of days.
-MERprods Specify the supported Trellix products for which MER should be
collected.
-MERlogs Number of days of Application Logs, System Logs, and Security
logs, to collect. Specify -1 to avoid collection for any of the event
logs.
-MERsanitize Remove IP address, MAC address, Domain names, and Computer
names from the MER result file.
-MERCSanitize [1 | 2] [RegEx filename | Text to sanitize] Use this switch to
sanitize the MER logs from a regex file or from a regex argument.
26
Running SSSO in Air-gapped Environments
1. On systems connected to the internet, download the SSSO_1.0.0.xxx.zip from the
location provided by Trellix support.
5. Run the required playbook to collect data. (See Playbook Execution Workflow (Running the SSSO
Playbooks on a Host) section in SSSO GUI Workflows).
Data Collection when endpoint security process is consuming high CPU on the system
The playbook monitors the CPU usage of a specified endpoint security process for a specified threshold
and collects PerfCounters, Procmon, AMTrace and MER data if there is a breach in threshold.
In case a customer wants to use a different CPU threshold value, you can now specify the threshold by -
threshold:<percentage> command.
Details
Step1:
27
Expected result: SSSO awaits the CPU utilization to breach the 40% threshold.
Step2:
Description: Reproduce the issue and wait for CPU to breach the threshold value and then initiate a full
on-demand scan for Trellix Endpoint Security on the system.
Expected Result: When a full on-demand scan is triggered, the CPU utilization exceeds the threshold
limit. SSSO starts collecting the PerfCounters, Procmon, and AMTrace data. In the end the MER logs are
collected.
Step3:
28
ePO Endpoint Deployment Kit (EEDK)
How to create an EEDK package for ePO deployment?
A one stop solution to deploy SSSO via Trellix ePO is through an EEDK package. To use it, follow the steps
below:
1. Upload the package (MFEDIAG_EEDK.zip) to master repository in ePO.
2. Select from the list of systems or groups in the system tree and create a Run Client Task Now
task by clicking Actions Agent Run Client Task Now.
3. In the Run Client Task Now window, select the product as Trellix Agent, task type as Product
Deployment and Create New Task.
4. On the Run Client Task Now page, select the Target Platform and then select the product as
Trellix Diagnostic Tool 1.0.0.xxx.
5. Provide the command line option -acceptThirdPartyEULA and provide the respective
command-line argument to run the playbook.
-acceptThirdPartyEULA -play <Playbook_Name.yml>
6. Provide appropriate command-line options required to run the specific playbook.
For example, if you are running “ENS_Process_HighCPU_Level1.yml” to collect data for
mcsheild.exe at a CPU threshold of 40 %, The total command line would be
29
9. Click Run Client Task Now.
11. If there are any SSSO failures, you can check SSSO_Updater.log and SSSO.log in
%ProgramData%\McAfee\SSSO.
Note: Please note that Playbooks with LiveKD.exe and NotMyFault.exe references cannot be run from
Trellix ePO.
You can run only one instance of SSSO at a given time. If you try to run multiple instances, you get the
below error:
Unsupported OS failure
SSSO can run on OS versions listed in Supported operating systems section. Running SSSO on older OS
versions will throw an unsupported OS error.
30
Powershell version is less than 3.0
For all workflows except Collect MER Workflow, PowerShell version should be 3.0 or greater. Otherwise,
you will get the following prompt.
Also, if "Exploit Prevention" is enabled, then "Execution Policy Bypass in PowerShell" rule should not be
configured to 'Block'.
Validation Failed
This error would appear if any unauthorized files were present in SSSO folders. Please check the logs to
understand the failure path and reason.
SSSO cannot be run from case sensitive paths. Please note that all the folders in the path should be case
insensitive. If any of the folders in path is case sensitive, you will get the below error:
31
Playbook Execution failed
Playbook execution could have failed for many reasons. Some common cases would be
32
Playbooks for Data Collection
There is a set of playbooks that comes with the package which can be used for data collection.
Note:
1. All playbooks require that the minimum PowerShell version to be 3.0 or above. If "Exploit
Prevention" is enabled, then "Execution Policy Bypass in PowerShell" rule should not be
configured to 'Block'.
2. For All Memory related playbooks, If ENS/VSE is installed, Self-protection must be disabled before
executing the playbook.
Default Playbooks
Common Playbooks
Basic:
33
UI_Generic_Procmon This playbook launches procmon and opens a dialog No
for the user. After reproducing the issue it saves the
procmon trace and initiates mer.exe in UI mode.
Basic:
Advanced:
34
Template_ProcessCrashWithWER This playbook demonstrates to monitor No
the event of a process crash (-
procname:<processname.exe>) using
Windows Error Reporting (WER).
Collects: process dump & MER.
35
Default path is as per
Windows 10 Debugging
Tools.
Windows Development
Kit (WDK) needs to be
installed to run
poolmon.exe. Make
sure the path to
PoolMon.exe is correct,
as different version of
WDK may have
different path. Current
path is as per Windows
10 Development Kit
36
Data Loss Prevention Endpoint Playbooks
Basic:
37
Endpoint Security Playbooks
Basic:
38
ENS_SlowAppStartup_Random This playbook monitory for process (- No
procname:<processname.exe>), if it is
taking long time for startup.
Collects: Procmon & AMTrace.
Advanced:
39
ENS_Process_HighCPU_level1 This playbook monitors the CPU usage Windows Assessment
of specified ENS process (- and Deployment Kit
procname:<processname.exe>) for a (ADK) needs to be
specified threshold (- installed to run
threshold:<value>). wpr.exe. Select
Collects : PerfCounters, Procmon, "Windows
AMTrace & MER if there is a threshold Performance Toolkit"
breach. during ADK setup.
Make sure the path to
umdh.exe is correct, as
different version of
Windows Assessment
and Development kit
tool may have different
path. Current path is as
per Windows 8
Assessment and
Development kit Tool.
40
ENS_BootLogging_Stop_Manual This playbook has to be executed after No
ENS_BootLogging_Start_Manual, once
the machine is up after the reboot.
Collects: Procmon & Amcore logs
during the boot time.
Advanced:
Basic:
41
MACC_Loglevel This playbook sets the required log levels for a No
specified MACC module.
42
Advanced:
MACC_HighCPU This playbook monitors the CPU usage of specified Windows Assessment
MACC process (-procname:<processname.exe>) for a and Deployment Kit
specified threshold (-threshold:<value>). (ADK) needs to be
Collects: Procdump, Procmon, WPR & MER if there is installed to run
a threshold breach. wpr.exe. Select
"Windows
Performance Toolkit"
during ADK setup.
Make sure the path to
umdh.exe is correct, as
different version of
Windows Assessment
and Development kit
tool may have different
path. Current path is as
per Windows 8
Assessment and
Development kit Tool.
MACC_Process_MemoryLeak This playbook is useful in detecting memory leaks for Windows Development
a given MACC process. Kit (WDK) needs to be
Collects: Procmon, Procdump & Poolmon over a installed to run
period of time. poolmon.exe. Make
43
sure the path to
PoolMon.exe is correct,
as different version of
WDK may have
different path. Current
path is as per Windows
10 Development Kit
Creating your own playbook is not a difficult task. The playbook.yml file under the doc folder helps create
a new playbook with very little effort. There are few things to remember while creating the playbook:
Outline your use case in simple tasks. Breakdown what is needed and whenever needed
Follow the conventions as described in playbook.yml
Avoid tab spaces in the playbook. The following is an FAQ from the official YAML website
(https://yaml.org/faq.html)
"Tabs have been outlawed since they are treated differently by different editors and tools. And since
indentation is so critical to proper interpretation of YAML, this issue is just too tricky to even attempt.
Indeed, Guido van Rossum of Python has acknowledged that allowing TABs in Python source is a
headache for many people and that were he to design Python again, he would forbid them."
The orchestrator supports playbooks of well-known workflows including product-specific workflows. The
user specific or product-specific workflows can reduce the time for auto remediation or effective data
collection. The orchestrator also supports building of custom playbooks based on the needs. The playbook
also supports invoking of different tools based on time or events or external triggers.
44
45