Web Gateway shows high CPU use, traffic slowness, and the maximum connection limit reached

Technical Articles ID: KB95284
Last Modified: 2022-03-22 18:17:47 Etc/GMT

Environment

Web Gateway 10.2.5 and earlier, 9.2.1 and earlier

Problem

Web Gateway shows one or more of the following issues:

High CPU use and slowness of operation:
- Access to webpages is delayed.
- You see a sudden increase (sometimes within minutes) for CPU Utilization in the dashboard.
The number of connections suddenly increases, up to the configured upper limit. This rise causes a delay for new client connections.
The proxy no longer accepts new connections. Even with usual number of connections or fewer, no new client connections are accepted. Clients receive an error message that the proxy isn’t available.

Cause

Special HTTP2 connections lead to stuck threads in the mwg-core process.

Solution

This issue is fixed in Web Gateway 9.2.17 and 10.2.6, and is included in all subsequent releases.

Workaround

To avoid this issue, disable HTTPv2 traffic.

Create a rule at the top of your rule set with following settings:

Criteria: always
Action: continue
Event: Proxy Control

You can also import the attached rule set Disable HTTP2.xml:

Download the attached rule set file Disable HTTP2.xml.
Select Policy, Rule Sets.
Click Add, Top-Level Rule set, Import rule set from Rule Set Library.
Click Import from file, Browse, choose the file Disable HTTP2.xml and Open.
Click OK. The new rule set Disable HTTP2 is now added to the bottom of your rule sets.
Move the new rule to the top of the rule set tree.
Click Save Changes.

Related Information

You can verify if your proxy is running in the named bug by checking the following indicators:

Create a feedback file while the problem occurs.
Download the feedback file to your system, open the archive, and open the file running_mwg-core.txt using a text editor.
Check for any of the following three indicators:

IMPORTANT: You might see all three indicators, or only one or two of them.
- Search for workingqueuelengths
  You see a single line showing active queue length such as:
  ### /opt/mwg/bin/mwg-core -S workingqueuelengths total = 144172; individual lengths = 0, 0, 0, 0, 0, 0, 0, 144172
  In this example, you see 8 queues where only one is busy with a large number. This series of zeros shows that the threads are stuck, and only one queue is handling all connections.
- Search for /opt/mwg/bin/mwg-core -S connections.
  You see a list of all current connections on the proxy. View the Status column and see if any are in the status Created as in the example below:
  
  ### /opt/mwg/bin/mwg-core -S connections
```
There are currently 147999 client connections (2022-01-07 14:23:38)
 
```
```
Creation Time       | Protocol |  Cust ID | <truncated> | Status | In use | URL
2022-01-07 14:08:24 |     HTTP |          | <truncated> | Created |     0 | 
2022-01-07 14:07:51 |     HTTP |          | <truncated> | Created |     0 | 
2022-01-07 14:21:44 |     HTTP |          | <truncated> | Created |     0 | 
2022-01-07 14:08:12 |     HTTP |          | <truncated> | Created |     0 | 

		
```
  If many connections are in status Created, it’s an indication that threads are stuck.
- Search for HTTP2 in the Threads section.
  
  As an example, here’s a finding in thread 123. Verify if #3 is CHTTP2FlowControl and #1 shows in futex_wait:
  
  Thread 123 (Thread 0x7f5c9e6e6700 (LWP 3170)):
  #0 0x00007f5d3fa08d29 in syscall () from /lib64/libc.so.6
  #1 0x00007f5d09e89362 in futex_wait(unsigned int volatile*, int, unsigned int) () from /opt/mwg/plugin/libProxy.so
  #2 0x00007f5d0a0a0d15 in CCriticalSection::GetLocked() () from /opt/mwg/plugin/libProxy.so
  #3 0x00007f5d09c51fde in CHTTP2FlowControl::SendHeader(CHTTPRequestResponse&, TFirstLine&, CScopedLockWithValue<std::pair<std::_Rb_tree_const_iterator<std::pair<CString const, std::pair<CString, eCharEncoding> > >, std::_Rb_tree_const_iterator<std::pair<CString const, std::pair<CString, eCharEncoding> > > > >, bool, bool, std::multimap<CString, std::pair<CString, eCharEncoding>, CString::TLessThanForMaps, std::allocator<std::pair<CString const, std::pair<CString, eCharEncoding> > > >&) () from /opt/mwg/plugin/libProxy.so
  #4 0x00007f5d09c4a974 in CHTTP2FlowControl::SendHeader(CHTTPRequestResponse&, bool, bool, std::multimap<CString, std::pair<CString, eCharEncoding>, CString::TLessThanForMaps, std::allocator<std::pair<CString const, std::pair<CString, eCharEncoding> > > >&) () from /opt/mwg/plugin/libProxy.so
  #5 0x00007f5d09c5d259 in CHTTPRequestResponse::SendRequestHeader() () from /opt/mwg/plugin/libProxy.so
  #6 0x00007f5d09c5ea3f in CHTTPRequestResponse::HandleEvent(CEvent&) () from /opt/mwg/plugin/libProxy.so
  #7 0x00007f5d09f38047 in AEventSerialHandler::ProcessEvent(CEvent&, bool) () from /opt/mwg/plugin/libProxy.so
  #8 0x00007f5d09f4d302 in CWorkingThreadPool::DoWork() () from /opt/mwg/plugin/libProxy.so
  
  This report is another indicator that the HTTP2 traffic is stuck.

An alternative indicator is a full receive queue. Check this queue by running the command netstat -alnp from the command line interface:

Log on to the Web Gateway command line interface via SSH.
Type netstat -anlp | less and press Enter.

Check the listener socket, which is listening on the proxy port (example 0.0.0.0:8080 and 0.0.0.0:9090):

Active Internet connections (servers and established)

	Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
	tcp        5      0 10.12.2.10:9090         0.0.0.0:*               LISTEN      1067/mwg-core       
	tcp        0      0 127.0.0.1:4710          0.0.0.0:*               LISTEN      1527/java           
	tcp        0      0 10.12.2.10:2121         0.0.0.0:*               LISTEN      1067/mwg-core       
	tcp      129      0 10.12.2.10:8080         0.0.0.0:*               LISTEN      1067/mwg-core       
	tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      2119/named          
	tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1935/sshd         
	tcp        0      0 127.0.0.1:12344         0.0.0.0:*               LISTEN      1354/mwg-coordinator

Here you can see the number 129 for the receive queue. Any number between 0 and 128 is fine. A higher figure indicates the receive queue is full and the proxy affected by the known issue.

Attachment

Disable HTTP2.xml
7K • < 1 minute @ broadband

Affected Products

Languages:

This article is available in the following languages:

Knowledge Center

Web Gateway shows high CPU use, traffic slowness, and the maximum connection limit reached

Environment

Problem

Cause

Solution

Workaround

Related Information

Attachment

Affected Products

Languages:

Title

Title

Knowledge Center

Web Gateway shows high CPU use, traffic slowness, and the maximum connection limit reached

Environment

Problem

Cause

Solution

Workaround

Related Information

Attachment

Affected Products

Languages:

Choose your region

Title

Title