Minimum data collection steps for Endpoint Security issues

Tools

SSSO is a tool that facilitates data collection using multiple standalone tools out of a single interface. SSSO uses predefined playbooks and triggers to automate the collection. For more information, see KB92519 - Self-Service Supportability Orchestrator data collection tool.
AMTrace is an internal tool to collect logging data from AMCore (last updated October 11, 2018). To use AMTrace to collect logging data from AMCore:
1. Prepare AMTrace:
  1. Download the zip package ENSDataCollect.zip from the "Attachment" section of this article.
  2. Extract the contents to the Desktop.
2. Run AMTrace:
  1. Click Start, type cmd.exe in the Search bar, right-click cmd.exe from the list, and click Run as administrator.
  2. When you're ready to start a trace, use the command option below that the relevant data collection section requires.
    
    NOTE: The following are the paths to the AMTrace.exe file locations:
    - C:\Users\username\Desktop\ENSDataCollect\AMTracex86
    - C:\Users\username\Desktop\ENSDataCollect\AMTracex64
    AMTrace command options:
    - To use the AMTrace onboot option, run the following command:
      AMTrace.exe -b onboot -m 4GB
      This command instructs the tool to begin a trace at the next boot.
      NOTES:
      
      The "GB" is case-sensitive. This example limits the log size to 4 GB. 10 MB is the minimum accepted value, and 2 GB is the default, if not specified.
      
      This option doesn't support the alternative logging modes described below.
    - To use AMTrace with the now option, run the following command:
      AMTrace.exe -b now -m 4GB
      
      IMPORTANT: The AMTrace now uses the rollover option by default.
      
      This command instructs the tool to begin a trace immediately, and to limit the log size to 4 GB per .etl file. When the log reaches 4 GB, a new log is created. Each log is appended with _1, _2, and so on, until you stop the trace, the user logs off, or you shut down the system.
      
      NOTE: The "GB" is case-sensitive.
    - To use AMTrace without the rollover option:
      Choose an appropriate logging method for the issue you want to record:
      AMTrace.exe -b now -m 4GB -L stop
      AMTrace.exe -b now -m 4GB -L circular
      The stop mode creates a trace session that stops logging once the size limit is reached.
      The circular mode creates a trace session that logs to a single file. After reaching the maximum size, older events are overwritten.
      
      NOTE: The "L" and "GB" are case-sensitive.
  3. Stop the trace and save the log. Use the following command:
    AMTrace -e
  4. When possible, AMTrace tries to automatically rename the resulting ETL files to include the start time and stop time of the logging in of the file name. For example, amtrace_20200704.010203-010305.etl indicates that the logging began on 2020-07-04 (July 4, 2020) at 1:02:03, and continued until 1:03:05.
    If AMTrace is unable to rename the file when logging stops, it's still possible to rename the file manually with another AMTrace command:
    AMTrace.exe --datestamp *.etl
    This command accepts wildcards (* or ?) to reference multiple characters or a single character, respectively. This command renames the specified file or files to include the start and stop times in the file name. It doesn't affect files that already have the datestamp added.
To confirm whether an AMTrace is in progress, run the following command to list any active traces:
AMTrace -q

For a demonstration of how to collect AMTrace data using this procedure, see the video Endpoint Security Data Collection AMTrace:
GFlags allows you to enable and disable advanced internal system diagnostic and troubleshooting features. You can run GFlags from a command prompt window or use its graphical user interface dialog box. It's most often used to turn on indicators that other tools track, count, and log.
MER collects event logs, file version details, files, process details, and registry details from products installed on your computer. Technical Support uses data collected to resolve problems. For more information, see KB59385 - How to use MER tools with our products.
Process Monitor is a tool from Microsoft that monitors and displays all file system activity on a Windows operating system in real time. Use the tool in system administration, computer forensics, and application debugging. How to use Process Monitor:
1. Prepare Process Monitor:
  1. Download Process Monitor from the Process Monitor downloads page.
  2. Extract Procmon.exe to the Desktop.
2. Run Process Monitor:
  1. When you're ready to start Process Monitor, use the option below that the relevant data collection section requires.
    - To immediately start Process Monitor:
      1. Run Procmon.exe; it automatically starts to capture process information.
      2. To stop Process Monitor, press Ctrl+E or click File and deselect Capture Events. Press Ctrl+E again to resume data collection.
      3. To save the log, click File, Save... (select All Events and use the native PML format).
    - To enable the Process Monitor boot logging option if needed by the relevant data collection section, perform the steps below:
      1. Open the Process Monitor console.
      2. Click Options.
      3. Click Enable Boot Logging.
      4. Click OK on the pop-up window. The next time a reboot occurs, a boot trace log is created.
      5. To save the log, run Process Monitor again and click File, Save... (select All Events and use the native PML format).
For a demonstration of how to collect Process Monitor data using this procedure, see the video Endpoint Security Data Collection Procmon.
ProcDump is a command-line utility used to monitor an application for CPU spikes. It generates crash dumps during a spike that an administrator or developer can use to determine the cause of the spike.
PerfMon is a tool that administrators can use to examine how programs, running on their computers, affect the computer's performance. Use the tool in real time to analyze how running programs affect system performance. You can also use this tool to collect log file information for system performance data analysis later.
PoolMon displays data that the operating system collects about memory allocations from the system paged and nonpaged kernel pools, and memory pools used for Terminal Services sessions. PoolMon groups the data by pool allocation tag. Microsoft Technical Support uses that information to find kernel mode memory leaks.
VMware converter is a free utility from VMware that helps convert Windows and Linux-based physical systems to VMware virtual machines. You can also use it to convert third-party image formats such as backup images and other virtual machines to VMware virtual machines. Use this tool to create virtual machines to provide to Technical Support for troubleshooting.
WinDbg is a Microsoft-distributed debugger for the Windows operating system. Use this tool to debug user mode applications, device drivers, and the operating system itself in kernel mode. It has a graphical user interface and is more powerful than Visual Studio Debugger.

Windows Performance Recorder (WPR) is a performance recording tool from Microsoft that's based on Event Tracing for Windows (ETW). It records system events that you can then analyze by using Windows Performance Analyzer (WPA). To use WPR, perform the steps below:

Click Start, type cmd.exe in the Search bar, right-click cmd.exe from the list, and click Run as administrator.
Type wprui.exe and press Enter to start WPR.

NOTE: If the program doesn't run or isn't found, you must first install it. WPR is part of the Windows Performance Toolkit. It's available from the Windows SDK or the Windows Assessment and Deployment Kit:
- For Windows SDK, see the Windows SDK downloads page.
- For the Windows Assessment and Deployment Kit, see the Windows Assessment and Deployment Kit downloads page.

Choose to use a Performance Scenario and other settings, as recommended in the following table:

Performance Issue	Performance Scenario	Detail Level	Logging Mode	Profiles to Include	Number of Iterations
Slow boot or logon	Boot	See below	File	First-level Triage, CPU usage, File I/O activity, Minifilter I/O Activity	At least 1
High CPU usage	General	See below	File	First-level Triage, CPU usage, File I/O activity, Minifilter I/O Activity	N/A
Application is slow or unresponsive	General	See below	File	First-level Triage, CPU usage, File I/O activity, Minifilter I/O Activity	N/A

WPR data is collected to allow for deeper analysis of the problem. The "Minifilter I/O Activity" option is one of the most important and is located under the "Scenario Analysis" section.

The use of WPR places extra strain on the system, which can change or mask the original problem that you want to investigate. Collect two data sets with different detail levels. Use the Light setting to show the issue, and the Verbose setting to allow a data set suitable for deeper analysis. Capture at least 30 seconds.

When possible, collect a WPR log without the issue while you perform the same task, for comparative purposes. A data set without ENS present is needed to establish the benchmark for expected performance.

For a demonstration of how to collect WPR data with this procedure, see the video Endpoint Security Data Collection - Windows Performance Recorder.

The following sections describe what data to collect for each type of issue:

Verify whether ENS debug logging is enabled

For instructions, see KB91797 - Enable debug logging to troubleshoot Endpoint Security issues.

Slow boot, startup, or logon

Perform the steps in this section if the symptoms are any of the following:

Slow boot or startup
Slow logon

SSO Playbook:

NOTE: Collect a WebMER from the SSSO utility immediately after the playbook is completed.

Playbook		ENS_BootLogging_Start_Auto.yml
Prerequisites			PowerShell version 3.0 Exploit Prevention signature "Execution Policy Bypass in PowerShell" disabled, or configured to not block ENS debug logging enabled
Tools SSSO Executes				Process Monitor AMTrace
Notes	This playbook configures Process Monitor and AMTrace to collect boot trace logging registered ENS_BootLogging_Stop_Auto.yml as a scheduled task on the next system reboot. You must manually reboot the system after executing this task. Check the SSSO logging for instructions. On reboot, ENS_BootLogging_Stop_Auto.yml executes and collects the boot logs.

Manual Data Collection:
Data collection steps for AMTrace and Process Monitor:

Start AMTrace with the onboot option.
Start Process Monitor and enable the boot logging option.
Reboot the system.
Reproduce the issue.
Log on to the system.
Stop AMTrace and save the log.
Open Process Monitor and save the boot log.
Collect a MER using the SSSO utility.

Data collection steps for Windows Performance Recorder:

Run WPR.
Configure the Boot Performance Scenario.
Start the capture.
Reboot the system.
Reproduce the issue.
Log on to the system.
Allow the WPR: Boot Trace to finish.
Capture the saved ETL files.
Collect a MER using the SSSO utility.

Slow performance (reproducible)

Perform the steps in this section if the symptoms are reproducible and are any of the following:

Slow application startup
Slow application performance
Slow system performance

SSSO Playbook

Playbook		ENS_Process_HighCPU_Level1.yml
Prerequisites			PowerShell version 3.0 Exploit Prevention signature "Execution Policy Bypass in PowerShell" disabled, or configured to not block ENS debug logging enabled Windows Performance Recorder installed
Tools SSSO Executes				Process Monitor AMTrace WPR PerfMon WebMER
Notes	This playbook requires that you provide the target process name and a CPU threshold to trigger the data collection.

Manual Data Collection:
Data collection steps for AMTrace and Process Monitor:

Start Process Monitor.
Start AMTrace with the now option.
Reproduce the issue.
Stop AMTrace and save the log.
Stop Process Monitor and save the log.
Collect a MER using the SSSO utility.

Data collection steps for WPR:

Run WPR.
Configure the General Performance Scenario.
Start the trace.
Reproduce the issue.
Stop the trace.
Capture the saved ETL file.
Collect a MER using the SSSO utility.

Slow performance (random)

Perform the steps in this section if the symptoms occur randomly and are any of the following:

Slow application startup
Slow application performance
Slow system performance

SSSO Playbook

Playbook		ENS_Process_HighCPU_Level1.yml
Prerequisites			PowerShell version 3.0 Exploit Prevention signature "Execution Policy Bypass in PowerShell" disabled, or configured to not block ENS debug logging enabled Windows Performance Recorder installed
Tools SSSO Executes				Process Monitor AMTrace WPR PerfMon WebMER
Notes	This playbook requires that you provide the target process name and a CPU threshold to trigger the data collection.

Manual Data Collection:
Data collection steps for AMTrace:

Start AMTrace with the rollover option.
When the issue occurs, stop AMTrace and save the log.
Collect a MER using the SSSO utility.

Data collection steps for WPR:

Run WPR.
Configure the General Performance Scenario with Memory as the Logging Mode.
Start the trace.
Reproduce the issue.
Save the trace as soon as possible after you reproduce the issue.
Capture the saved ETL file.
Collect a MER using the SSSO utility.

System hang, deadlock, or bug check (blue screen)

Perform the steps in this section if the symptoms are any of the following:

System hang or deadlock
System bug check (blue screen)

SSSO Playbook (System Hang or Deadlock):

NOTE: Collect a WebMER from the SSSO utility immediately after the playbook is completed.

Playbook		ENS_Process_HighCPU_Level3.yml
Prerequisites			PowerShell version 3.0 Exploit Prevention signature "Execution Policy Bypass in PowerShell" disabled, or configured to not block ENS debug logging enabled System configured for a full or kernel memory dump.
Tools SSSO Executes				NotMyFault
Notes	You need to configure the "Process Name" variable for this playbook (for example, mcshield.exe). The respective process name is monitored for a breach of CPU consumption threshold (default is 40%). A system crash is triggered once that threshold is passed.

Manual Data Collection:
Data collection steps for a system hang or deadlock:

Configure the system to create a full memory.dmp. See KB56023 - Create a memory dump for analysis by Technical Support.
Configure the system to allow for a keyboard crash. See this Microsoft article to force a keyboard crash.
Create the dump file when the issue occurs. Generally, the longer you can wait before you generate the dump file, the easier it is to identify the hang condition in the dump.
If the system is in an accessible state, collect a MER using the SSSO utility.

Data collection steps for a system bug check:

Configure the system to create a full memory.dmp. See KB56023 - Create a memory dump for analysis by Technical Support.
Collect the full dump file when the system bug check (blue screen) occurs.
If the system is in an accessible state, collect a MER using the SSSO utility.

Application hang, deadlock, or crash

Perform the steps in this section if the symptoms are any of the following:

Application hang or deadlock (not responding and doesn't recover)
Application crash

SSSO Playbook

Playbook		Template_ProcessHang.yml
Prerequisites			PowerShell version 3.0 Exploit Prevention signature "Execution Policy Bypass in PowerShell" disabled, or configured to not block ENS debug logging enabled ENS Self-Protection disabled
Tools SSSO Executes				Process Dump WebMER
Notes	You must provide the target process, or the process experiencing the hangs, to execute this playbook. If you need assistance in determining the specific process name, contact Technical Support.

Manual Data Collection:
Data collection steps for an application hang or deadlock:

Download ProcDump from the ProcDump downloads page.
Extract ProcDump to the Desktop.
Open an administrative command prompt, and change the directory to C:\Users\username\Desktop\Procdump.
Run the following command: procdump -ma <process name>
Collect the created dump file, which is in the Procdump folder.
Collect a MER using the SSSO utility.

Data collection steps for an application crash:

If the crashing process is an ENS process, disable ENS Self-Protection.
Download ProcDump from the ProcDump downloads page.
Extract ProcDump to the Desktop.
Open an administrative command prompt, and change the directory to C:\Users\username\Desktop\Procdump.
Run the following command: procdump -ma -e <process name>

NOTE: The -e switch instructs ProcDump to generate a dump the next time the process crashes.
Wait for the process to crash again.
Collect the created dump file, which is in the Procdump folder.
Re-enable ENS Self-Protection.
Collect a MER using the SSSO utility.

Memory leak (user or kernel)

Perform the steps in this section if the symptoms involve a User Mode or Application memory leak. Collect three (3) User Mode or Application crash dumps for analysis.

NOTE: Memory leak data collection needs to be performed over time, just as the memory leak exhibits its behavior over time. For the best data collection results, it’s recommended to reboot the system, and then start the following data collection. This sequence allows the data set to show the existence of the memory leak, over time, as it manifests on the system.

SSSO Playbook (User Mode Leak)

Playbook		ENS_Process_MemoryLeak.yml
Prerequisites			ENS Self-Protection must be disabled ENS debug logging Windows Development Kit (WDK) must be installed on the system Windows Debugging Tools must be installed on the system (Included in the WDK installation) PowerShell version 3.0 ENS Exploit Prevention signature "Execution Policy Bypass in PowerShell" rule shouldn't be configured to "Block"
Tools SSSO Executes				GFlags Taskkill UMDH PoolMon AMTrace Process Monitor WebMER
Notes	This playbook requires that you provide a target process name for tracking its memory footprint over time. To execute this playbook correctly, ENS Self-Protection must be disabled during testing.

Manual Data Collection (User Mode Leak):

Download ProcDump from the ProcDump downloads page.
Extract ProcDump to the Desktop.
Identify the process name that's leaking memory.
Enable a stack trace on the leaking process. See KB91252 - How to enable a stack trace using the gflags.exe utility.
Wait for the suspect process to show high memory usage.
Open an administrative command prompt, and change the directory to C:\Users\username\Desktop\Procdump.
Run the following command: procdump -ma <process name>
Collect the created dump file, which is in the Procdump folder.
Repeat the steps and collect three (3) User Mode or Application crash dumps for analysis.
Disable the stack trace on the process once all crash dump files are collected. See KB91252 - How to enable a stack trace using the gflags.exe utility.

Manual Data Collection (Kernel Mode Leak):
Perform the steps in this section if there's a suspected kernel memory leak involving one of our processes.

Familiarize yourself with PoolMon and PerfMon usage and configuration described in KB74951 - How to troubleshoot high memory use on systems.
Configure the system to create a full memory.dmp. See KB56023 - Create a memory dump for analysis by Technical Support.
Configure the system to allow for a keyboard crash. See this Microsoft article to force a keyboard crash.
Reboot the system reported to show a memory leak.
Use the configuration for PoolMon and PerfMon outlined in KB74951 - How to troubleshoot high memory use on systems. Start the PoolMon and PerfMon data collection.
Wait for the system to show high memory usage.
Stop PoolMon and PerfMon, and collect the resulting data.
Force the system to perform a bug check while the high memory usage is still exhibited.
Collect the memory dump.

Issues related to Device Guard or Credential Guard

Perform the steps in this section if the symptoms involve Device Guard or Credential Guard.

Collect the appropriate ENS data for the experienced symptom, as outlined in this article.
Also, collect an ETW trace using the following command, executed in an administrative command prompt:

@echo off
ECHO These commands enable tracing:
@echo on
logman create trace "base_DeviceGuard" -ow -o c:base_DeviceGuard.etl -p "Microsoft-Windows-DeviceGuard" 0xffffffffffffffff 0xff -nb 16 16 -bs 1024 -mode Circular -f bincirc -max 4096 -ets
@echo off
echo
ECHO Reproduce your issue and enter any key to stop tracing
@echo on
pause
logman stop "base_DeviceGuard" -ets
@echo off
echo Tracing has been captured and saved successfully at c:base_DeviceGuard.etl
pause

ENS components fail to install

Perform the data collection steps in this section if one or more components of ENS fail to install.

NOTE: In most situations, it's ideal for ENS installation data collection to be done using the local installation path. This configuration isolates any potential failures to the ENS installer itself, and not task execution or network errors. If data collection must be done from ePO-based deployments, work with Technical Support to make sure this configuration is considered when the data is reviewed.

SSSO Playbook (Local Installation)

Playbook		ENS_InstallationFailure.yml
Prerequisites			PowerShell version 3.0 Exploit Prevention signature "Execution Policy Bypass in PowerShell" disabled, or configured to not block ENS debug logging enabled
Tools SSSO Executes				Process Monitor AMTrace WebMER System File Checker (SFC)
Notes	This playbook watches for the process spawn of SetupEp.exe, and begins collecting data at the beginning of an ENS local installation.

SSSO Playbook (ePO Deployment)

Playbook		ENS_InstallationFailure_ePO.yml
Prerequisites			PowerShell version 3.0 Exploit Prevention signature "Execution Policy Bypass in PowerShell" disabled, or configured to not block ENS debug logging enabled
Tools SSSO Executes				Process Monitor AMTrace WebMER System File Checker (SFC)
Notes	This playbook watches for the process spawn of SetupCC.exe, and beings collecting data at the beginning of an ENS ePO-based deployment.

Manual Data Collection:

NOTE: Make sure that you collect the data during a local installation of ENS. Troubleshoot each module as a separate product.

Download and unzip the standalone package from the Product Downloads site.
Start Process Monitor.
Start AMTrace with the rollover option.
Re-create the issue. Run the local installation (setupEP.exe) as administrator and select the single module you're troubleshooting.
Stop AMTrace and save the log.
Stop Process Monitor and save the log.
Collect a MER file (run as Administrator) using the SSSO utility.

ENS isn't running or third-party DLL injection

Perform the steps in this section if the symptoms are any of the following:

The status of ENS is "Endpoint Security Platform isn't running!"
Third-party DLL injection

SSSO Playbook

Playbook		ENS_Process_HighCPU_Level1.yml
Prerequisites			PowerShell version 3.0 Exploit Prevention signature "Execution Policy Bypass in PowerShell" disabled, or configured to not block ENS debug logging enabled Windows Performance Recorder installed
Tools SSSO Executes				Process Monitor AMTrace WPR PerfMon MER
Notes	Set the parameters to Process Name "mfeesp.exe" and Threshold 0%. Once the playbook is executed, immediately open and close the ENS Console to reproduce the issue.

Manual Data Collection:

Start Process Monitor.
Start AMTrace with the now option.
Open and close the ENS Console to re-create the issue.
Stop AMTrace and save the log.
Stop Process Monitor and save the log.
Collect a MER file (run as Administrator).
Export and collect a copy of the assigned Endpoint Security Common Options policy.

Issues that involve the Threat Intelligence Exchange (TIE) module

Perform the steps in this section if the symptoms are related to TIE:

Collect the appropriate data based on the symptoms outlined in this article.
Also, collect the TIE Server log on the TIE Server appliance at /var/McAfee/tieserver/logs/tieserver.log.

Issues that involve the ENS Firewall module

Perform the steps in KB90662 - Troubleshoot an application/network traffic when using ENS Firewall.

Back to top

Knowledge Center

Minimum data collection steps for Endpoint Security issues

Environment

Summary

Related Information

Attachment

Affected Products

Languages:

Title

Title

Knowledge Center

Minimum data collection steps for Endpoint Security issues

Environment

Summary

Related Information

Attachment

Affected Products

Languages:

Choose your region

Title

Title