Category: Networking

Network security policy installation failure fix

Some times your network policy installations on Check Point devices might fail.  For me it happened after updating a gateway cluster to the “latest and greatest R77” version. I was unable to push the policy and I was getting the “/opt/CPSFWR77CMP-R77/conf/policy-name.pf”, line 912700: ERROR: target <fw-name> is prohibited” error message.

In order to see what is actually causing the error you will need to log in to the management server via SSH. Go in to “expert mode” and look at what is on the line that the error message is pointing at.

So basically, to fix the issue in my case/work around, I did the following:

  1. Logged in to the security management server in expert mode
  2. Opened up the policy file from the place it was complaining about with the less command. (Hints on how to go to a specific line can be found stackoverflow topic here)
  3. In my case on that specific line I saw a list of DPD(dead peer detection for IPSec VPN) peers which hinted that I should try and disable DPD
  4. Logged in to the management server using smart dashboard, removed permanent tunnel ticks on VPN’s relating to the GW cluster with the issue and tried installing the policy.
  5. Policy successfully installed..

After that I reported the bug to Check Point support and they confirmed the issue..

Enabling DPD on VPN instead of tunnel_test on R77 gateway

To keep VPN tunnels alive Check Point uses by default it’s proprietary tunnel_test protocol. In order to get it working with 3rd party vendors it isn’t enough to have the partner device set as an “Interoperable device” and set the tunnel keep alive method on your gateway object as DPD. You also need to set the peer gateway’s tunnel keep alive method as DPD, because by default it is still set to tunnel_test.

To change the keep alive methods you need to do the following as described on Check Point’s website here:

  1. In GuiDBedit, go to Network Objects > network_objects > <gateway> > VPN > tunnel_keepalive_method.
  2. For the Value, select a permanent tunnel mode.
  3. Save.
  4. Install policy on the gateways.

ConfigSync issue upon resource pool member IP address change

Well as it turn’s out changing a node’s IP address in clustered environments doesn’t go that smoothly as one would expect. Not only that, F5 have made a annoyingly complex procedure of something that simple as changing one back end servers IP address. What I mean by that is, that in order to change the IP address of a node you actually have to delete the node and then re-create it. But when you have one node running a ton of services on different ports and is a part of a large amount of resource pools, etc you have to remove it from all of the resource pools before you can delete it. And then after creating the node again you have have to put it back in the pools you need it to be in.

As it turns out in a clustered environment when you do the aforementioned procedure and then try to sync the cluster member settings manually it will fail with an error message saying something like this:

0107003c:3: Invalid pool member modification. An IP address change from (192.168.1.1) to (192.168.1.2) is not supported.

So in order to avoid that error you need to sync the devices after you have finished all the deletion steps of the node and after you have done the config sync you may proceed with the creation of the node on the new IP address.

If you are already in the state where it is refusing to sync, well what helps is deleting the troublesome node on the secondary device also and then performing the config sync.

Check Point IPS signature updates through URL filtering blade with HTTPS inspection enabled

According to Check Point’s documentation you should let your SMS and Smart Dashboard clients get access to the internet freely without actually intercepting the traffic it self. There might be some cases where you wouldn’t like that idea too much and might still want to inspect it to restrict HTTPS connectivity of those hosts. After a bit of messing about, I managed to get IPS signature update through URL filtering working.

Basically when you try and update IPS signatures or have your Smart Dashboard try and connect to Check Point via URL filtering it will fail with errors relating to the certificates. In other words if you have followed the regular Check Point guidelines on setting up “App control/URL filter” and set the system to trust your gateway’s Certificate Authority, it will still fail. The reason behind it being the fact that Smart Dashboard client uses it’s own trust list not the system wide certificate store. The trust list file is located in “C:\Program Files (x86)\CheckPoint\SmartConsole\R77.30\PROGRAM\ca-bundle.crt” and when you add your own gateway’s CA certificate there after you have already started Smart Dashboard it will work until you restart Smart Dashboard. As I found out that file is actually being downloaded off the SMS every time Smart Dashboard starts up.

Fortunately the filename on the SMS is actually the same as it is in the SmartDashboard folder so when running the standard linux find command you see that there are quite a few instances of it:

/opt/CPsuite-R77/fw1/bin/ca-bundle.crt
/opt/CPinfo-10/bin/data/ca-bundle.crt
/opt/CPcvpn-R77/var/ssl/ca-bundle.crt
/opt/CPcvpn-R77/template/var/ssl/ca-bundle.crt
/opt/CPshrd-R77/database/downloads/CA_BUNDLE/1.0/1.0/ca-bundle.crt
/var/opt/CPsuite-R77/fw1/conf/SMC_Files/asm/ca-bundle.crt
/var/opt/CPshrd-R77/conf/ca-bundle.crt

Now make a backup of /var/opt/CPsuite-R77/fw1/conf/SMC_Files/asm/ca-bundle.crt and /var/opt/CPshrd-R77/conf/ca-bundle.crt files and your gateway’s CA certificate in PEM format to the end. After doing that restart Smart Dashboard and check if the ca-bundle file has your CA certificate in it and if communication to Check Point actually works.

Just as a reminder, the URL’s that need to be enabled for IPS signature to work can be found on CP support site here: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk112635

CheckPoint VPN with failed connection not showing up in SmartView Tracker

Scenario description

You might have an old existing VPN with some partner which to a gateway that is not under your control. That peer gateway used to be also a CheckPoint device but get’s exchanged for some other 3rd party vendor firewall with out prior notification. When the peer gateway gets changed, the key exchange seems to work, but connection’s fail and nothing seems to be showing up in SmartView tracker although connection logging is on.

Finding out what is happening and trying to fix it

After seeing no traffic in SmartView tracker, you log in to the gateway and might look at tcpdump to see if there is actually any traffic coming from the peer gateway. After seeing there is actual traffic coming you should try a few kernel level debug commands on VPN to get some info on whats happening. For example run “fw ctl zdebug -m VPN all” and you might see something like this in the output:

 [cpu_1];[fw4_0];fw_log_drop_conn: Packet <dir 1, 192.168.200.100:0 -> 172.32.200.100:0 IPP 50>, dropped by do_inbound, Reason: decryption failed

Well this still isn’t saying too much, but at least is giving a hint. Next lets take a look at the secure XL debug, maybe it gives some more info. To get the output prepare the debug first by using the following commands:

[Expert@HostName:0]# fw ctl debug 0
[Expert@HostName:0]# fw ctl debug -buf 32000
[Expert@HostName:0]# sim dbg -m vpn all
[Expert@HostName:0]# sim dbg -m mgr + vpn
Now start the debug:
fw ctl kdebug -T -f > /tmp/debug.txt

Let it run for a bit and close it with ctrl+c. Go through the log file, you might find something like this:

;11Jul2017 13:14:57.017362;[cpu_1];[SIM-xxxx]vpn_ipsec_decrypt: Packet does not match block size (xx xx xx);

;11Jul2017 13:14:57.017362;[cpu_1];[SIM-xxxx]vpn_decrypt: IPSEC decryption failed;

Well this is odd. You contact the peer gateway’s technical contact asking if have they changed anything lately and find out that they changed their firewalls..

Fixing the issue

To remedy the issue create a new gateway object that is of the “Interoperable Device” type and has all the same settings/parameters as the previous the peer gateway object that was with the “Check Point gateway” type. Change the peer GW object in the VPN community to the newly created “Interoperable device” and install policy. And the traffic starts flowing again.

As it seems Check Point is handling VPN tunnels differently between Check Point devices and Interoperable devices.

Trunk between Cisco Catalyst switches and HP Aruba 5400zl R2

When creating a trunk link between a Cisco Catalyst switch and a HP 5400zl R2 switch, it would be a good idea to have it work in LACP mode.  It is quite simple, you just create the bond on both switches and add the VLAN’s you want on to the trunk link. And here is a little configuration example to show how it goes.

On the HP 5400 side just use the following commands:

trunk A24,B24 trk1 lacp
vlan 10 tagged trk1
vlan 20 tagged trk1
vlan 4000 untagged trk1

On the Cisco Catalyst side it requires a few more commands, but isn’t difficult either. Just add interfaces to a channel group and add the VLAN’s on to the channel group, by doing the following:

interface range GigabitEthernet1/0/24,GigabitEthernet2/0/24
switchport mode trunk
channel-protocol lacp
channel-group 1 mode active
interface Port-channel1
switchport trunk native vlan 4000
switchport trunk allowed vlan 10,20
switchport mode trun

802.1x Authentication on Cisco Catalyst switches

If any one is wondering how to configure 802.1x authentication on Cisco switches, here is a little list of commands that should help you. I am not going to cover the hassle of configuring NPS service on Windows domain server at the moment, but maybe later on if I get the time for it.

Basics of what you need to do

  1. Configure the radius servers on the switch
  2. Set the switches authentication mode to aaa new-model and configure aaa authentication it self
  3. Configure the VLANS you want to use (not going to cover creating a VLAN here..)
  4. Configure the ports you want the authentication to be required on

The configuration

So lets start configuring the switch (using the command line interface). First log in to the switch a. (I seriously hope you are using SSH not telnet..)

Configuring the radius servers

You should replace the IP addresses ports and passwords to match the ones you are going to use. PS password will be in plain text in the configuration unless you have service password encryption turned on. I am configuring the switch to use 2 radius servers, since its always a good idea to have more than one of them.. In case one fails the switch will try the other one to authenticate the user.

radius server name-of-first-server
 address ipv4 10.0.0.101 auth-port 1812 acct-port 1813  
 key  YourRadiusPasswordHere

radius server name-of-second-server
 address ipv4 10.0.0.102 auth-port 1812 acct-port 1813
 key  YourRadiusPasswordHere
aaa new-model 
aaa group server radius MyRadiusServers
 server name name-of-first-server
 server name name-of-second-server

Enabling authentication

Set the user authentication service to use your previously configured radius group radius servers

aaa authentication dot1x default group MyRadiusServers
aaa authorization network default group MyRadiusServers

Enabling user authentication on the ports

After configuring the radius part on the switch it’s time to enable authentication on the ports you want. As an example I enabled them on ports 1-12 leaving the others alone.

interface range gi1/0/1-12
switchport access vlan 5
switchport mode access
authentication event fail action authorize vlan 2
authentication event no-response action authorize vlan 2
authentication port-control auto
dot1x pae authenticator
dot1x timeout tx-period 2

The “switchport access vlan” command sets the VLAN that the user will be put in to incase of a successful authentication, in the case of this example it’s vlan 500.  In other words, it sets the ports default vlan.
The commands starting with “authentication event” set the VLAN where the user ends up in case of authentication failure.
Dot1x timeout it’s not a mandatory command, but a nice thing to set if you want to use authentication fail to send people to some guest network. The thing what I noticed was that the clients that don’t authenticate them selves some times ended up with the auto configure addresses – windows just gave up on trying to get access to the network, when this time out was not set.

R77 database revision control unspecified error

It is always a good idea to have database revision control on just in case. But keep in mind that the comment field in it doesn’t support all sorts of characters. As it turns out using commas (“,”) inside comments are illegal and ends up with the revision creation failing and SmartDashboard client giving you an “unspecified error” message. Just remove the comma or any other “strange symbol” from it and it works fine. In other words remember to keep the comments short and simple, with out special characters..

CheckPoint Gaia embedded sic reset

If you happen to have a CheckPoint 1400 series firewall hooked up to your central management and for some reason need to reset the sic communication between the firewall and management then this command will help (it’s not the same as in the full blown Gaia OS):

set sic_init password Y0urS!cPassw0rd

In R77 getting the whole connections table in human readable format

When you need to get the connections table out in human readable format from your CheckPoint R7x firewall and get some sort of idea on how many connections there are between certain hosts and who have the most this line might help you.

fw tab -t connections -u -f|grep Direction|cut -d';' -f3,5-7|sort -n|uniq -c|sort -rn > connections`date +"%Y-%m-%dT%H-%M"`.txt

Depending on the connections table size it may take quite a bit of time and I would suggest doing in on the standby node not to spend the active devices CPU time on it.