NSX DFW logging to syslog server

One of the strongest features of NSX is that every single VM is protected by the Distributed Firewall (DFW). If logging is enabled the logs of the DFW are written in a file called “dfwpktlogs.log” on the local ESXI host, where the VM is hosted. (on pre 6.1 installation, these logs are written in the “vmkernel.log”.

It is however very easy to collect those logs on a centralised syslog server, which makes troubleshooting or just onderstanding the working of NSX much easier. Below I’ve written down the steps to quickly setup a centralised syslog server and how to collect the DFW logs.

Setting up the syslog server
First make sure you have a linux machine up and running, which will act as syslog server, in my case it is a Debian 8 machine. RSyslog which we will use for this example is installed by default (as it is on most distros), but not enabled.

Make sure you have the right permission and edit the file /etc/rsyslog.conf.

Uncomment the following lines;

So it will result in:

Add the following lines before the “Global Directives”, so all logging will be collected and place within a subdirectory /var/log/rsyslog.

Start the rsyslog deamon by running

This is enough on the syslog side, you can check with “netstat -nau” if your server is listening on udp port 514.

For more about the rsyslog server you can use this link, it was my resource.

Configure vSphere
Make sure SSH is enabled on your ESX hosts and SSH into it. Use the following commands to open up the firewall and enable the syslog server.

Set the syslog server, don’t forget to change it with your own IP of the syslog server.

Check if the logs are received by the syslog server, by listing the rsyslog directory on the syslog server.

Before the firewall logs will appear, you need to enable logging on the distributed firewall rules. You only have to do this once for every action you want to have logged, since it is euhm…a distributed firewall.

Go to you distributed firewall and click on the pencil in the action column on the corresponding line.

NSX DFW log-action

Don’t forget to publish the changes.
DFW Publish ChangesNow generate some traffic to or from one of the machines living on the host, where you just enabled remote syslog. If you look in your “.\rsyslog\host” directory on the syslog server, you should see a file “dfwpktlogs.log” appear.

You can easily view this file by cat or tail.

If you want to see the logging of multiple files and follow it when new lines are written, I find it easy to use a tool called, “xtail”. You can simply install it from the Debian repositories.

You can easily start xtail by passing the multiple files you want to follow. The below example is an SSH session started from a VM (192.168.1.11) on host ESX03 to a VM (192.168.1.12) on host ESX04. As you can see, the first hit on the DFW is the outgoing traffic on ESX03 and a than the incoming traffic on ESX04. It is also worth noticing that these VM’s are living on the same subnet and still passing the firewall, pretty cool 🙂

 

Facebooktwittergoogle_plusredditpinterestlinkedinmailby feather

3 thoughts on “NSX DFW logging to syslog server

  1. Hi,

    I also use a centralized syslog-ng server to collect ESX / dfw logs. I noticed that some packets get DROPed despite being explicitly allowed by filtering rules. It seems to be caused by some sort of protocol analysis detecting an anomaly. When such DROPs occur, letters are appended to the end of the log line, e.g :

    dfwpktlogs: INET match DROP domain-c7/1015 IN 40 TCP xxx.xxx.xxx.xxx/50997->xxx.xxx.xxx.xxx/443 FA

    Do you by any chance know where I can find the meaning of those status codes such as A / RA / FA / FPA, etc. ?

    Regards,
    Julien.

    • Hi Julien,

      Not entirely sure, but since it is essentially based upon IPTables, I think the flags are the same, so;
      A = Ack
      SA = Syn Ack
      RA = Reset Ack
      FA = Finalizing Ack
      FPA = Full Packet Analysis -> Which (I think) you only will see when traffic is steered to a service VM.

      • Hi Rob,

        Thanks for your answer. I’ve been in contact with VMWare support about another issue and they confirmed you’re correct about the TCP flags 🙂

        Kind regards,
        Julien.

Leave a Reply

Your email address will not be published. Required fields are marked *

*