Monday, February 6, 2012

pfSense Log Analysis with Splunk

Customizing Splunk to parse pfSense logs

For those Basement PC Techs (BPCT) out there that want to send their pfSense traffic to Splunk or have tried and realized that Splunk doesn't automatically parse the logs as it should. Well I got good news for you, I have create the necessary configuration that will allow Splunk to not only parse your data but parses the data the way you want to see your firewall traffic in Splunk by the following fields:

Source IP
Source Port
Destination IP
Destination Port
Action (Pass or Block) 

The pfSense logs for each firewall event is split into two lines when it is sent to Splunk which Splunk doesn't automatically recognize. By editing two configuration files you can configure Splunk to parse the pfSense event as one so it can be parsed correctly. The two files that we will create/edit are the props.conf and transforms.conf. Each file will need to be created (or edited if one already exist) in the following location:


The props.conf file is were we will configure Splunk to recognized the multi-line events sent from pfSense as one. If you want more detail on what is the purpose of the props.conf file please see the Splunk website:

  1. Create / Edit a props.conf file in $SPLUNK_HOME/etc/system/local/
  1. Cut and Paste the following into the props.con file:




MUST_NOT_BREAK_AFTER = pf: .* rule ([-\d]+\/\d+)\(.*?\):

MUST_BREAK_AFTER = pf: .* (<|>) +(\d+\.\d+\.\d+\.\d+)\.?(\d*)\:

REPORT-pf2 = pf2

  1. Save the file


The transforms.conf file is were we will configure Splunk to parse the pfSense events received into the fields we want to see. If you want more detail on what can be done with the transforms.conf file please visit the Splunk website:

  1. Create / Edit a transforms.conf file in $SPLUNK_HOME/etc/system/local/
  1. Cut and Paste the following into the transforms.conf file:


REGEX= .* (?<action>pass|block) .* (?<protocol>TCP|UDP|IGMP|ICMP) .* (?<src_ip>(\d+\.\d+\.\d+\.\d+))\.?(?<src_port>(\d*)) [<|>] (?<dest_ip>(\d+\.\d+\.\d+\.\d+))\.?(?<dest_port>(\d*)): (.*)

  1. Save the file
  1. Reboot Splunk in order for the new changes to take affect.

Note: Although the above "REGEX" statement may shows on multiple lines it is actually all on one line. Ensure word wrap is off when you paste the text to your transforms.conf file or download the following transforms.conf from here.


  1. You rock, this answered my question spot on and got my data into a better format. Just wanted to throw you a thanks!

  2. Hi,
    I'm not quite sure how the output should be, however, even with the section added to the transforms.conf file I cant see the formatted data. It shows exactly like the raw data with the two merged lines and not the separate fields like I believe it should display. I'm on splunk 5.0.2.
    Thanks a lot for your help.

    1. Bruno,

      I provide a link to a transforms.conf file that has already been configured to process PFSense logs. Try that file.

    2. Hello Bruno,

      I believe that you are asking on how to display the logs better, its done by adding a command like table on your search so that it will display by fields that was extracted by the transforms given above.

      index=sample_index|table _time src_ip dest_ip protocol dest_port action for any other questions with field extraction in splunk

  3. Thanks! You should make a Splunk app to make this easier!


  4. The solution two get *1* line within PFSense and I have this working in pfsense 2.1

    Go to Diagnostics -> Edit File, open file /etc/inc/ and change the following:

    From: mwexec_bg("/usr/sbin/tcpdump -s 256 -v -l -n -e -ttt -i pflog0 | logger -t pf -p"); To: mwexec_bg("/usr/sbin/tcpdump -s 256 -v -l -n -e -ttt -i pflog0 | /usr/bin/sed -l -e 'N;s/\n //;P;D;' | logger -t pf -p");

    You have to reboot your pfsense to see the changes applied, and there no more split lines!

    NOTE: credits goes to: