All ePolicy Orchestrator events fail to parse and eventually get stuck in the Events folder
Technical Articles ID:
KB86011
Last Modified: 2023-07-24 04:50:41 Etc/GMT
Last Modified: 2023-07-24 04:50:41 Etc/GMT
Environment
ePolicy Orchestrator (ePO) 5.x
Problem
All ePO events fail to parse, and eventually get stuck in the Events folder.
TheEventParser.log records the errors below:
The
E #08888 EPOEVENTS epoevents.cpp(50): COM Error 0x80040E31, source=Microsoft OLE DB Provider for SQL Server, desc=Query timeout expired, msg=IDispatch error #3121
When you view the SQL Activity Monitor in the SQL Server Management Studio, you might find a query similar to the one below. The query shows that it's blocking many other queries, including insert event queries:
For more information about opening SQL Activity Monitor, see this Microsoft article.
Cause
This problem can be made worse when many events exist in the EPOEvents table. When an Event query is executed via a non-Global administrator account, ePO must run permission checks against the user account. This action is needed to verify that the user has group and product access for each of the queried events. This process is resource-intensive within SQL and can use large amounts of disk I/O, memory, and CPU resources.
If there are millions of events when ePO executes this type of query, it can cause the query to take a long time and prevent ePO from accessing the events table when it tries to insert events.
This problem can also occur if there are limited resources on the SQL Server, including the following:
If there are millions of events when ePO executes this type of query, it can cause the query to take a long time and prevent ePO from accessing the events table when it tries to insert events.
This problem can also occur if there are limited resources on the SQL Server, including the following:
- Insufficient memory installed on the SQL Server or insufficient memory allotted to SQL
- Poor disk I/O performance
- High CPU usage on the SQL Server
- Database fragmentation
Solution
The following best practices are recommended to resolve issues when many events exist and there are limited resources on the SQL Server.
- Purge unnecessary events from the
EPOEvents table. Accomplish this task using either an ePO purge task, or manually through SQL Server Management Studio.
For details, see KB68961 - How to find the top 10 threat events and purge events. - Add memory to the SQL Server* or see this Microsoft article. This article provides instructions about how to allot more memory for SQL use.
- Fine-tune the configuration of your SQL hardware for your environment, including hard disk space, CPU resources, and memory. See the relevant ePolicy Orchestrator Best Practices Guide for instructions.
- For environments with more than 10,000 nodes, make sure that SQL is installed on a physical server. For important sizing recommendations for ePO, see the ePO 5.10 Installation Guide.
- Look for database index fragmentation and rebuild indexes, if needed.
For the recommended maintenance plan for ePO databases, see KB67184 - REGISTERED - Recommended maintenance plan for ePolicy Orchestrator databases using SQL Server Management Studio.
NOTE: The referenced content is available only to logged in ServicePortal users. To view the content, click the link and log in when prompted.
Workaround
To find blocking queries within the SQL Activity Monitor, perform the following:
- Expand the Processes pane and find the column labeled Head Blocker. The column labels are truncated. You can see the full name by mousing over the column name or expanding the column size.
- Click the drop-down arrow on the Head Blocker column and select 1. This action lists the blocking query. If there's no number 1, and only the word All is listed, there's no blocking process. Also, you're not seeing the issue referenced in this article.
- Right-click the command listed in the Command column, and select Details to show the entire query.
If you see a query similar to the above that's blocking, you can try to kill the query. Right-click the query and select kill.
Confirm whether this action allows events to begin to parse. If killing the query temporarily resolves the issue, search theOrion log for an instance of the query found in the SQL Activity Monitor.
For assistance, see KB52369 - How to enable debug logging in the Orion.log.
If you find an instance of the query in the Orion log, you might identify what's generating this query. This type of query is most often a result of a query executed from a user dashboard within ePO. If so, you can try closing all open ePO consoles for all users, to see whether this action also temporarily resolves the issue.
Related Information
For product documents, go to the Product Documentation portal.
Affected Products
Languages:
This article is available in the following languages:
