Monday, February 6, 2012

pfSense Log Analysis with Splunk


Customizing Splunk to parse pfSense logs

For those Basement PC Techs (BPCT) out there that want to send their pfSense traffic to Splunk or have tried and realized that Splunk doesn't automatically parse the logs as it should. Well I got good news for you, I have create the necessary configuration that will allow Splunk to not only parse your data but parses the data the way you want to see your firewall traffic in Splunk by the following fields:

Source IP
Source Port
Destination IP
Destination Port
Protocol
Action (Pass or Block) 


The pfSense logs for each firewall event is split into two lines when it is sent to Splunk which Splunk doesn't automatically recognize. By editing two configuration files you can configure Splunk to parse the pfSense event as one so it can be parsed correctly. The two files that we will create/edit are the props.conf and transforms.conf. Each file will need to be created (or edited if one already exist) in the following location:

$SPLUNK_HOME/etc/system/local/
props.conf

The props.conf file is were we will configure Splunk to recognized the multi-line events sent from pfSense as one. If you want more detail on what is the purpose of the props.conf file please see the Splunk website: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

  1. Create / Edit a props.conf file in $SPLUNK_HOME/etc/system/local/
  1. Cut and Paste the following into the props.con file:

[syslog]


SHOULD_LINEMERGE = true

TRUNCATE = 0

MUST_NOT_BREAK_AFTER = pf: .* rule ([-\d]+\/\d+)\(.*?\):

MUST_BREAK_AFTER = pf: .* (<|>) +(\d+\.\d+\.\d+\.\d+)\.?(\d*)\:

REPORT-pf2 = pf2

  1. Save the file

transforms.conf

The transforms.conf file is were we will configure Splunk to parse the pfSense events received into the fields we want to see. If you want more detail on what can be done with the transforms.conf file please visit the Splunk website: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf


  1. Create / Edit a transforms.conf file in $SPLUNK_HOME/etc/system/local/
  1. Cut and Paste the following into the transforms.conf file:

[pf2]

REGEX= .* (?<action>pass|block) .* (?<protocol>TCP|UDP|IGMP|ICMP) .* (?<src_ip>(\d+\.\d+\.\d+\.\d+))\.?(?<src_port>(\d*)) [<|>] (?<dest_ip>(\d+\.\d+\.\d+\.\d+))\.?(?<dest_port>(\d*)): (.*)


  1. Save the file
  1. Reboot Splunk in order for the new changes to take affect.

Note: Although the above "REGEX" statement may shows on multiple lines it is actually all on one line. Ensure word wrap is off when you paste the text to your transforms.conf file or download the following transforms.conf from here.

6 comments:

  1. You rock, this answered my question spot on and got my data into a better format. Just wanted to throw you a thanks!

    ReplyDelete
  2. Hi,
    I'm not quite sure how the output should be, however, even with the section added to the transforms.conf file I cant see the formatted data. It shows exactly like the raw data with the two merged lines and not the separate fields like I believe it should display. I'm on splunk 5.0.2.
    Thanks a lot for your help.
    Bruno

    ReplyDelete
    Replies
    1. Bruno,

      I provide a link to a transforms.conf file that has already been configured to process PFSense logs. Try that file.

      http://www.basementpctech.com/content/download-transformsconf

      Delete
    2. Hello Bruno,

      I believe that you are asking on how to display the logs better, its done by adding a command like table on your search so that it will display by fields that was extracted by the transforms given above.

      index=sample_index|table _time src_ip dest_ip protocol dest_port action

      dalisayreagan@gmail.com for any other questions with field extraction in splunk

      Delete
  3. Thanks! You should make a Splunk app to make this easier!

    ReplyDelete

  4. The solution two get *1* line within PFSense and I have this working in pfsense 2.1

    Go to Diagnostics -> Edit File, open file /etc/inc/filter.inc and change the following:

    From: mwexec_bg("/usr/sbin/tcpdump -s 256 -v -l -n -e -ttt -i pflog0 | logger -t pf -p local0.info"); To: mwexec_bg("/usr/sbin/tcpdump -s 256 -v -l -n -e -ttt -i pflog0 | /usr/bin/sed -l -e 'N;s/\n //;P;D;' | logger -t pf -p local0.info");

    You have to reboot your pfsense to see the changes applied, and there no more split lines!

    NOTE: credits goes to: http://thwack.solarwinds.com/thread/54381

    ReplyDelete