How to set up Endpoint Security for Servers for reporting and troubleshooting
Technical Articles ID:
KB92389
Last Modified: 2022-05-02 04:19:25 Etc/GMT
Environment
Endpoint Security (ENS) for Servers 5.x
Summary
This article describes how to perform the actions below:
Set up ENS for Servers
Determine whether the scan tasks run as configured
Identify common issues with running scan tasks
Currently, there are no built-in reports for ENS for Servers in ePolicy Orchestrator (ePO). For example, there are no reports that show the following:
How many virtual machines (VMs) are eligible for the scan task
How many VMs get the scan task
How many VMs the scan task get executed and either pass or fail
ENS for Servers sends these details and other information to ePO, which is stored in the Smart Scheduler tables in the ePO database.
Contents
Click to expand the section you want to view:
The following information explains how and where the ENS for Servers extension stores different data in the ePO database. When you install the ENS for Servers extension in ePO, it creates six database tables in the ePO database:
AMPQClientTable
AMPQClientTasksTable
AMPQResourceInfoTable
AMPQSlotTable
AMPQTaskEnableDisableStatusTable
AMPQTimeZonesTable
Below is an explanation of what each of these tables store.
Table Name
Summary
Stored Properties
AMPQClientTable
This table holds the list of all VMs that Cloud Workload Security (CWS) provides.
This table holds the following properties of the client VMs:
EpoLeafNodeId (ePO leaf node ID of the individual VMs)
AgentGUID (Agent GUID of the individual VMs)
TimeZone (current time zone of the individual VMs)
PowerStatus (current power status of the individual VMs)
AgentVersion (current McAfee Agent (MA) installed version of the individual VMs)
AMPQClientTasksTable
This table holds the list of all client tasks that are selected on the Smart Scheduler Catalog page in ePO.
This table holds the following properties of the client tasks:
EpoTaskUid (unique ID of the ePO task)
TaskName (name of the task)
TaskType (type of the task)
ProductName (product name for which the task is created)
IsAMPQEnabled (whether the task is enabled in Smart Scheduler)
ePOTaskId (ePO task ID; it's a unique ID for different tasks selected)
NOTE:The most important property is IsAMPQEnabled. When you enable or select the task on the Smart Scheduler Catalog page, this property becomes 1. When you disable or deselect the task on the Smart Scheduler Catalog page, this property becomes 0.
NOTE:ENS for Servers supports the following client tasks for scheduling:
ENS Threat Prevention Policy Based On-Demand Scan (Full scan and Quick scan)
ENS Threat Prevention Custom On-Demand Scan
MA Product Update Task for AMCore Content Package
AMPQResourceInfoTable
This table holds the current state of the hypervisor. The CWS provides the hypervisor details. This table keeps updating each minute, and the details for each minute are appended to the table.
This table holds the following properties of the hypervisors:
ResourceId (hypervisor unique ID)
ResourceType (hypervisor resource type)
CurrentLoad (current CPU utilization of the hypervisor)
Threshold ("Threshold %" setting on the Smart Scheduler page)
Capability (the value depends on how the internal algorithm decides to trigger a batch of tasks on endpoints)
SampleTime (time stamp of each entry of the details that CWS gathers each minute)
TasksStarted (number of VMs on which the task starts at the specific SampleTime)
TasksRequested (number of VMs on which the task is requested at the specific SampleTime)
TasksEnded (number of VMs on which the task is completed at the specific SampleTime)
TasksRunning (number of VMs on which the task runs at the specific SampleTime)
AMPQSlotTable
This table holds the slot details selected on the Smart Scheduler page in ePO.
This table holds the following properties of the hypervisors:
DayOfWeek (can hold 1–7, denoting Monday to Sunday)
StartTime (start of each hour of the day)
EndTime (end of each hour of the day)
AMPQTaskEnableDisableStatusTable
This table holds the task status of each of the individual VMs that Smart Scheduler selects to run the task on.
This table holds the following properties of the individual VMs:
EpoLeafNodeId (ePO leaf node ID of the individual VMs)
EpoTaskId (task ID that gets assigned to the specific VMs)
IsStateUpdated (task enable state of the individual VMs)
LastRunStartTime (start time stamp of the last task run)
LastRunEndTime (end time stamp of the last task run)
Status (status of the task)
NOTE:The most important property is IsStateUpdated. When a task is enabled on the Smart Scheduler Catalog page, ePO checks for the VMs on which the specific task is assigned. After finding the list of the VMs, ePO sends a Data Channel message to all those VMs to let them know that the specific task is now AMPQ enabled. This message means that the VMs should stop scheduling the task according to ePO, and start scheduling the task according to Smart Scheduler. After getting this Data Channel message, when the MA on individual VMs sends an acknowledgment back to ePO, this property is set to 1. Otherwise, this property remains as 0.
NOTE:The Status property can have six different states:
0 (not yet requested)
1 (requested)
2 (running)
3 (passed)
4 (failed)
NULL (not yet considered for scheduling)
AMPQTimeZonesTable
This table holds the time zones with an ID.
This table holds the following property of the individual time zones:
TimeZoneId (a specific ID for all individual time zones available)
Below are examples of problems you might encounter while using ENS for Servers:
NOTE: Although your exact issue might not be listed here, this article aims to provide you with general guidance on how to analyze problems.
After enabling the task, none of the VMs trigger the task.
Out of the total VMs, some systems don't get scanned.
VMs weren't scanned this week, but were scanned last week.
You're unable to tell whether the scan result was PASS or FAIL.
Even after scheduling the task using Smart Scheduler, the VM CPU usage is up to 100%.
Even after scheduling the task using Smart Scheduler, the hypervisor CPU usage always stays high.
Owing to an ePO upgrade or restart, tasks are no longer getting scheduled or executed on the VMs.
The following steps provide an overview of how to set up ENS for Servers and trigger the tasks using Smart Scheduler. For detailed information, see the Endpoint Security for Servers Installation Guide and the Endpoint Security for Servers Product Guide. ENS for Servers is designed for virtual infrastructures. Examples of supported virtual infrastructures are VMware, Citrix, Hyper-V, AWS, and Azure. The systems must be part of a virtual infrastructure. ENS for Servers doesn't support physical systems.
Make sure that the systems (virtual, not physical) are managed with ePO. Also, make sure that the systems are properly grouped. Assigning the task to a group is easier than assigning the task to individual systems.
NOTE:The Smart Scheduler can only schedule tasks on managed virtual machines, and not on unmanaged systems.
Install the following extensions in the order listed below:
MA
ENS Platform
ENS Threat Prevention
Data Center Controller (DCC)
Cloud Connector extensions (for example, vSphere Connector, Citrix XenServer Connector, and Microsoft Hyper-V Connector)
NOTE:After installing the Connector extension, register the cloud account and wait for the sync to pass.
Smart Scheduler
Create a client task in the Client Task Catalog for Smart Scheduler to schedule the task on the selected systems. Smart Scheduler supports the following types of client tasks:
ENS Threat Prevention Policy Based On-Demand Scan (Full scan and Quick scan)
ENS Threat Prevention Custom On-Demand Scan
MA Product Update Task for AMCore Content Package
This overview process focuses on on-demand scan tasks. Either create a custom on-demand scan task, or use an existing policy-based "On-Demand Scan Full Scan" or "On-Demand Scan Quick Scan" task. If you create a custom on-demand scan task, make sure that you configure the settings below on the new task creation page on the Client Task Catalog page. If you use an existing policy-based scan task, make sure that you configure the settings below. Go to the ePO Policy Catalog, ENS Threat Prevention, On-Demand Scan, and edit the appropriate policy.
In the Scheduled Scan Options section, select only the Scan anytime option. Deselect the rest of the options in this section.
In the Account Options section, leave the credentials blank.
NOTE:In the Mozilla Firefox browser, the Username field gets automatically prepopulated. The task saves without filling in the Password field. If you fill in the credentials, it causes the task to fail on the endpoint.
Assign the task to the appropriate groups. Or, assign the task to individual VMs.
Configure Smart Scheduler. Go to the Smart Scheduler page in ePO under Menu, Configuration.
Select the time slots on which you want the task to get scheduled and executed on the systems. Smart Scheduler uses only the selected slots to schedule the task.
Provide a CPU Threshold % value for the hypervisor. Smart Scheduler controls the task scheduling based on the threshold value set here.
NOTE: Smart Scheduler, using its self-learning algorithm, schedules the task on the system in batches. Smart Scheduler schedules batches in a way that the hypervisor CPU utilization doesn't cross the threshold value that you set. But, you might still see the threshold breach a few times on the hypervisor, which Smart Scheduler controls in later batches. These breaches might be due to parameters and circumstances occurring on the systems that Smart Scheduler doesn't control. For example, there might be activity on the systems that have a high CPU usage, such as an ENS DAT update or large file copy.
Go to the Smart Scheduler Catalog page under Menu, Policy. You can see different MA and ENS tasks pre-populated. Select the appropriate task and save.
NOTE:You can select one or multiple tasks from the list. Smart Scheduler schedules the selected tasks according to the selected slots and the CPU threshold value set. The slots selected and the configured CPU threshold are common for all tasks. Smart Scheduler isn't yet designed to have different slots or CPU threshold values for different tasks.
The task is now ready for Smart Scheduler to schedule on the assigned systems, based on the time slots chosen.
The following steps explain how scan task scheduling works using the Smart Scheduler:
When you install the Smart Scheduler extension, it fetches all details of all VMs present in the DCC table and populates them in the AMPQClientTable table.
When you select the slots on the Smart Scheduler page, the AMPQSlotTable table gets updated.
When you select the task on the Smart Scheduler Catalog page, the Smart Scheduler checks the selected task and populates the AMPQClientTasksTable table and updates the property IsAMPQEnabled to 1.
Smart Scheduler checks the VMs on which the task is assigned and populates the list of VMs in the AMPQTaskEnableDisableStatusTable table.
The property IsStateUpdated in the AMPQTaskEnableDisableStatusTable table updates to 0 for the VMs.
Smart Scheduler communicates through ePO to all individual VMs through the Data Channel (DC) to stop the regular scheduling of the task and start scheduling according to Smart Scheduler.
When the MA on the VMs gets the DC message, it sends the acknowledgment to ePO.
After receiving the acknowledgment from the individual VMs, Smart Scheduler updates the property IsStateUpdated for those VMs to 1.
Smart Scheduler keeps querying DCC to get the hypervisor CPU utilization each minute and keeps updating the AMPQResourceInfoTable table.
When the VMs are updated with the property IsStateUpdated as 1, these VMs are queued for the next schedule.
The Smart Scheduler algorithm finds the VMs that are eligible for scheduling according to the time zone selected, time slots selected, and current CPU utilization of the hypervisor. When Smart Scheduler creates the queue, it creates batches out of this queue and sends the DC message to the respective VMs to trigger the task.
Smart Scheduler keeps updating the AMPQResourceInfo table each minute with the number of tasks Requested, Started, Running, and Completed on the hypervisor.
Before you start the analysis, gather the following data:
All Smart Scheduler database tables in CSV format.
NOTE:CSV format is important. If you collect the tables in text format, it's hard to sort and filter.
The EPOLeafNodeMTtable from the ePO database, which helps identify the VM names, using the EpoLeafNodeId.
The MDCC_VM_INFO table from the ePO database, which helps identify the VM power statuses and managed states.
The MDCC_VM_PROPERTY table from the ePO database, which helps identify the VM hypervisor names and hypervisor IDs.
The debug logs of the ePO Orion log with the Smart Scheduler tag enabled. For instructions to enable debug logging for the Orion log with the Smart Scheduler tag, contact Technical Support.
The MA debug logs from the individual VMs, if needed.
The on-demand scan activity log from the individual VMs, if needed.
After you collect the data above, create an XLS using all database tables. Use the XLS to perform the analysis.
Create an XLS that contains the following:
The EpoLeafNodeID, AgentGUID, PowerStatus, and AgentVersion of all VM entries mentioned in the AMPQClientTable table.
The NodeName of the above VMs from the EpoLeafNodeMT table.
The EpoTaskId, IsStateUpdated, LastRunStartTime, LastRunEndTime, and Status from the AMPQTaskEnableDisableStatusTable table.
To note the results for each task running on the individual VMs, create a Comments column.
After creating the XLS, add NA in all blank places. NA indicates that the specific property isn't considered for the specific VM. The Attachment section of this article contains a sample XLS for your reference.
The following example process uses a sample XLS, available in the Attachment section of this article, to analyze the tasks:
View the total number of VMs (no physical systems) listed. In this sample, the total number of VMs is 3602. This number is the list of VMs taken from DCC. So, these VMs are present in the Cloud Account after the connector extension syncs them.
Verify the managed state using the Agent Version column and Agent GUID column. The total number of VMs includes both managed and unmanaged VMs. Out of 3602 VMs, 1862 are managed and 1740 are unmanaged.
Verify the power status using the Power Status column. The total number of VMs includes both turned on and turned off VMs. Out of 3602 VMs, 3548 are turned on and 54 are turned off.
Use the ePO Task ID column to get the list of VMs on which the task is assigned. Out of 3602 VMs, the task is assigned on 1798 VMs. The remaining 1804 VMs don't have the task assigned. For Smart Scheduler analysis, the focus is mostly on the list of VMs on which the task is assigned. Filter the Task ID column by the VMs with the task assigned.
Investigate why VMs are turned off or unmanaged. Out of 1798 VMs on which the task is assigned, you can see a few VMs that are in a turned off state, and a few VMs that are in an unmanaged state. Filter the Power Status column and Agent GUID column to only VMs that are turned on and managed. Out of 1798 VMs, 1793 VMs are turned on and managed.
Check the IsStateUpdated column state of the VMs. As stated earlier in this article, the IsStateUpdated property shows whether the VM has enabled the Smart Scheduler based scheduling for the assigned task. Out of 1793 VMs, the task is enabled for Smart Scheduler based scheduling for 1757 VMs. For the remaining 36 VMs, the task isn't enabled for Smart Scheduler based scheduling. It's important to understand why the task isn't enabled on the 36 VMs. There might be multiple reasons, for example:
The VMs have a DC issue. As a result, ePO isn't able to send the DC message to enable the task for Smart Scheduler.
The VMs were recently turned off. The DC message never reached the VMs, and the connector sync hasn't yet happened after the VMs were turned off.
Out of 1757 VMs, filter on the LastRunStartTime property using the Category column. Select only the VMs that executed the task during the last week (in this example, from October 25 to October 28). Out of 1757 VMs, the task was successfully triggered on 1691 VMs during this period. On the remaining 66 VMs, the task didn't run during this period. Investigate the 66 VMs because the task is enabled on those VMs (IsStateUpdated is 1). Possible reasons that the VMs aren't considered for scheduling include the following:
The VMs were communicating with ePO while enabling the task, but communication was later stopped. The VMs might have a DC issue.
The VMs didn't get the slots to execute the task.
Out of the 1691 VMs, where the task was successfully triggered, use the Task Status column to understand the following for the VMs:
The task is yet to be triggered (State = 0)
The task is requested (Status = 1)
The task is still running (Status = 2)
The task passed (Status = 3)
The task failed (Status = 4)
Depending on the analysis, you can filter on the Comment column.
On the VMs where the task isn't executed, use the ePO debug logs, MA debug logs, and on-demand scan activity logs to further debug the issue.
