How to generate a Java Heap dump or thread dump to troubleshoot an ePO issue
Technical Articles ID:
KB73309
Last Modified: 2023-03-27 10:01:27 Etc/GMT
Environment
ePolicy Orchestrator (ePO) 5.x
Summary
Sometimes, you might encounter issues that require a Java Heap dump or Thread dump.
Examples include the following:
- The ePO server is out of memory.
- You see errors such as OutOfMemory in the Orion.log file on the ePO server.
- Tomcat is in an unresponsive or error state.
- Technical Support requests a Heap dump.
CAUTION: This article contains information about opening or modifying the registry.
- The following information is intended for System Administrators. Registry modifications are irreversible and could cause system failure if done incorrectly.
- Before proceeding, Technical Support strongly recommends that you back up your registry and understand the restore process. For more information, see the Microsoft Windows registry information for advanced users article.
- Do not run a REG file that is not confirmed to be a genuine registry import file.
Problem
The Orion.log file records a Garbage Collection (GC) error:
java.sql.SQLException: java.lang.OutOfMemoryError: GC overhead limit exceeded
Solution
1
Generate a Java Heap dump when ePO encounters an OutOfMemory error in the Orion.log file on the ePO Application Server. To obtain the dump, change how the Tomcat and Java services start:
Set the Java sub key to -XX:+HeapDumpOnOutOfMemoryError:
- Click Start, Run, type regedit, and then click OK.
- Navigate to and expand the following registry key:
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\MCAFEETOMCATSRV5100\Parameters\Java
- Select the Java subkey.
- In the right pane, double-click the Options value and append the value data below to the bottom:
-XX:+HeapDumpOnOutOfMemoryError
- Click Start, Run, type services.msc, and then click OK.
- Restart the ePO Application Server service.
- If the error occurs again, run the Minimum Escalation Requirements (MER) tool for ePO. Also, make sure that the Java Heap dump is included in the MER results.
For details, see KB72895 - How to collect a Minimum Escalation Requirements log for ePolicy Orchestrator and McAfee Agent.
NOTES:
- By default, the Heap dump is created in the root of the ePO installation directory.
- The file is called Java_<pid>.hprof, where the <pid> is the Process ID for the currently running tomcat process (either tomcat7.exe or tomcat9.exe).
- On 64-bit operating systems, the (*.hprof) dump file might be written to the c:\windows\SysWow64\ folder rather than the root of the ePO installation folder. It can also be located in the server\bin folder within the ePO installation directory.
Solution
2
Generate a Java Heap dump when ePO encounters a GC overhead limit exceeded error in the Orion.log file. To obtain the dump, change how the Tomcat and Java services start:
Set the Java sub key to -XX:+HeapDumpBeforeFullGC:
- Click Start, Run, type regedit, and click OK.
- Navigate to and expand the registry key below:
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software Foundation\Procrun 2.0\MCAFEETOMCATSRV5100\Parameters\Java
- Select the Java subkey.
- In the right pane, double-click the Options value and append the value data below to the bottom:
-XX:+HeapDumpBeforeFullGC
- Click Start, Run, type services.msc, and click OK.
- Restart the ePO Application Server service.
- If the error occurs again, run the Minimum Escalation Requirements (MER) tool for ePO. Also, make sure that the Java Heap dump is included in the MER results.
For details, see KB72895 - How to collect a Minimum Escalation Requirements log for ePolicy Orchestrator and McAfee Agent.
NOTES:
- By default, the Heap dump is created in the root of the ePO installation directory.
- The file is called Java_<pid>.hprof, where the <pid> is the Process ID for the currently running tomcat process (either tomcat7.exe or tomcat9.exe).
- On 64-bit operating systems, the (*.hprof) dump file might be written to the c:\windows\SysWow64\ folder. Instead of the root of the ePO installation folder. It can also be located in the server\bin folder within the ePO installation directory.
Solution
3
Generate a Java Heap dump on demand using JMAP:
- Download JDK 1.6 or later from the Java Downloads page on Oracle and install it on your ePO server.
- Download PSExec from Sysinternals and install it on your ePO server.
- Wait for the error condition to occur or perform any needed steps to reproduce it.
- Open a command prompt with administrator rights and navigate to the directory that contains the psexec.exe.
- Run the following command:
psexec -s "<path to JDK installation directory>\bin\jmap.exe" -dump:format=b,file=heap.bin <PID for Tomcat>
Example:
psexec -s "C:\Program Files (x86)\Java\jdk1.7.0_01\bin\jmap.exe" -dump:format=b,file=heap.bin 3408
The Heap dump file can be found in:
- 32-bit:
C:\Windows\System32\
- 64-bit
C:\Windows\SysWOW64\
Solution
4
Generate a Java thread dump using jstack:
- Download JDK 1.6 or later from the Java Downloads page on Oracle and install it on your ePO server.
- Download PSExec from Sysinternals and install it on your ePO server.
- Wait for the error condition to occur or perform any needed steps to reproduce it.
- Open a command prompt with administrator rights and navigate to the directory that contains the psexec.exe.
- Run the following command:
psexec -s "<Path to JDK installation folder>\bin\jstack" -l <PID for Tomcat> > <Path to where you want the output file>\jstack_output.txt
Example:
psexec -s "C:\Program Files (x86)\Java\jdk1.7.0_01\bin\jstack.exe" -l 3408 > C:\jstack_output.txt
Solution
5
Generate a Java Heap Dump or Java Thread Dump on-demand using a browser command:
NOTE: The following only works with ePO 5.10 Update 7 or later.
- Access the ePO server.
- Open a web browser.
- Paste one of the following commands, and press Enter:
To capture a Heap Dump, type https://server_name:8443/remote/core.heapDump
To capture a Thread Dump, type https://server_name:8443/remote/core.threadDump
NOTES:
- Replace server_name with the actual name of the ePO server.
- Change the port number when needed. The default provided is 8443.
- The Heap dump "hprof" file is located in the folder ePO_Install_folder>\Server\Logs
|