Jump to content


Photo

[RESOLVED] Erroneous NG application definitions causing high CPU load


  • Please log in to reply
10 replies to this topic

#1 Bernhard Patsch

Bernhard Patsch
  • Barracuda Team Members
  • 103 posts

Posted 27 January 2017 - 05:24 PM

UPDATE 2017-02-01

 

Important Advisory: Issue caused by Pattern Update

 

Summary: On Jan 27th, 3 pm UTC new application definitions were released for the Barracuda NextGen Firewall F-Series. The included Content-Pattern file had corrupted data in it. Due to this defect, the parsing of the definitions failed on the firewall, causing some processes to loop and as a result effecting a high CPU load. The effect was more pronounced on smaller appliance models and may not have been noticed on larger appliance with many CPU cores.

The following processes have been affected:

  • trans7
  • acpffwdrule
  • appidctrl

The Barracuda Network Security Team quickly withdrew the corrupted definitions at 5:45 pm UTC, but the update pattern delivery process does not (for security reasons) allow for the delivery of executables, nor can an arbitrary executable (or script) be invoked throughout pattern update processing. Therefore, Barracuda had no possibility to fix the issue from remote by restarting the affected processes. Even after downloading new, fixed update definition files, these processes remained in a dysfunctional state.

 

Impact: Even though generally at least one trans7 process was locking up, the Firewall service kept running using the current firewall ruleset. The service did at no point in time unload the ruleset, or switch to "fail open " or "fail close" mode. However, subsequent firewall ruleset changes (done when the box was in this state) were not processed and written to kernel space, which means that the active configuration of the firewall service did not change, even though a ruleset change was performed through the configuration interface. This error condition is only resolved through a restart of the firewall service, or by applying the provided hotfix.

 

Affected Firmware: Barracuda NextGen Firewall F-Series Firmware Version 6.2.x or 7.0.x

 

Mitigation: Affected Firewalls cannot be remediated automatically but need to have a hotfix installed. Customers who notice the described symptoms should IMMEDIATELY install the following hotfix. Note that this hotfix works for 6.2.x and 7.0.x

 

We apologize for any inconvenience caused by this issue. We are constantly evaluating our quality assurance processes and will take appropriate measures to immunize our systems against similar incidents in the future.


Edited by Bernhard Patsch, 01 February 2017 - 02:16 PM.
Added latest information on ruleset issues and hotfixes


#2 Thomas Geppel RHI

Thomas Geppel RHI
  • Members
  • 4 posts

Posted 30 January 2017 - 03:13 AM

FYI: critical side-effect:
this fix solved also our problem that Firewall rules with configured user authentication were not working anymore.
After installing the fix, the rules were matching again. (Thanks to your Support team which really fast helped me here!)

Please if possible it would be great that you pro-actively inform your partners/customers due to this critical site-effect.



#3 Stefan Hora

Stefan Hora
  • Barracuda Guru
  • 129 posts

Posted 30 January 2017 - 06:55 AM

There is another side effect:

 

New Rules or Changes to Rules (which don't) have a user authentication) did not match in the forwarding Firewall.

 

The Hotfix helped also.

 

FYI: this HF does not change any code, it only updates the downloaded (broken) application files and then restarts the box Firewall Service.



#4 Pieter Rubens

Pieter Rubens
  • Members
  • 17 posts

Posted 31 January 2017 - 05:03 AM

Sorry Barracuda but the communication around this issue could have been a lot better... 

 

Rules not matching, fw's crashing, this is a big issue to a lot of our customers



#5 Thomas Heymans

Thomas Heymans
  • Members
  • 11 posts

Posted 01 February 2017 - 05:36 AM

Good work - we wasted several support hours troubleshooting rules not matching.

Thanks for posting a tiny message on your forum instead of sending out a proper e-mail communication.  :angry:



#6 Guido Kramer

Guido Kramer
  • Members
  • 17 posts

Posted 01 February 2017 - 09:05 AM

Hello,

 

one question: one of our customers has the following problem: the firewall systems do not have direct access to the Internet and the Application Control is not activated. It could be possible to connect via a http proxy but the dns resolution does not work.

 

So, how is it possible that the corrupted pattern files were installed on these firewall sytems?



#7 Alois Klingler

Alois Klingler
  • Barracuda Team Members
  • 21 posts

Posted 02 February 2017 - 04:31 AM

Hello Guido,

 

if those boxes are managed by a CC the CC downloads the patterns and delivers them to the boxes.

 

Best regards

Alois Klingler


Alois Klingler
Barracuda Networks

#8 Guido Kramer

Guido Kramer
  • Members
  • 17 posts

Posted 02 February 2017 - 11:08 AM

Hello Alois,

 

the firewall systems are not managed by a CC.

 

 

Regards,

Guido



#9 dinhmh

dinhmh
  • Members
  • 7 posts

Posted 10 February 2017 - 03:56 AM

I'm used F280 with Firmware is 7.0.0, but frequently restart the system with error: Received -3 or EMAXLOAD: LOAD Average High too.

Is this case caused by Pattern Issue Update?
 
Thanks,


#10 Robert Czymoch

Robert Czymoch
  • Members
  • 57 posts

Posted 10 February 2017 - 05:14 PM

Unclear.

 

Does a simple firewall restart fix the issue or do you have to apply the hotfix?



#11 Bernhard Patsch

Bernhard Patsch
  • Barracuda Team Members
  • 103 posts

Posted 21 February 2017 - 05:57 AM

Unclear.

 

Does a simple firewall restart fix the issue or do you have to apply the hotfix?

 

Robert,

 

It is recommended to apply hotfix.

The hotfix ensures that potentially affected processes are restarted in the right order.

 

regards,

Bernhard