pfSense Log Analysis with Splunk

Customizing Splunk to parse pfSense logs

For those Basement PC Techs (BPCT) out there that want to send their pfSense traffic to Splunk or have tried and realized that Splunk doesn't automatically parse the logs as it should. Well I got good news for you, I have create the necessary configuration that will allow Splunk to not only parse your data but parses the data the way you want to see your firewall traffic in Splunk by the following fields:

Source IP

Source Port

Destination IP

Destination Port

Protocol

Action (Pass or Block) 

The pfSense logs for each firewall event is split into two lines when it is sent to Splunk which Splunk doesn't automatically recognize. By editing two configuration files you can configure Splunk to parse the pfSense event as one so it can be parsed correctly. The two files that we will create/edit are the props.conf and transforms.conf. Each file will need to be created (or edited if one already exist) in the following location:

$SPLUNK_HOME/etc/system/local/

props.conf

The props.conf file is were we will configure Splunk to recognized the multi-line events sent from pfSense as one. If you want more detail on what is the purpose of the props.conf file please see the Splunk website: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

1. Create / Edit a props.conf file in $SPLUNK_HOME/etc/system/local/

2. Cut and Paste the following into the props.con file:

[syslog]

SHOULD_LINEMERGE = true

TRUNCATE = 0

MUST_NOT_BREAK_AFTER = pf: .* rule ([-\d]+\/\d+)\(.*?\):

MUST_BREAK_AFTER = pf: .* (<|>) +(\d+\.\d+\.\d+\.\d+)\.?(\d*)\:

REPORT-pf2 = pf2

3. Save the file

transforms.conf

The transforms.conf file is were we will configure Splunk to parse the pfSense events received into the fields we want to see. If you want more detail on what can be done with the transforms.conf file please visit the Splunk website: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf

1. Create / Edit a transforms.conf file in $SPLUNK_HOME/etc/system/local/

2. Cut and Paste the following into the transforms.conf file:

[pf2]

REGEX= .* (?<action>pass|block) .* (?<protocol>TCP|UDP|IGMP|ICMP) .* (?<src_ip>(\d+\.\d+\.\d+\.\d+))\.?(?<src_port>(\d*)) [<|>] (?<dest_ip>(\d+\.\d+\.\d+\.\d+))\.?(?<dest_port>(\d*)): (.*)

3. Save the file

4. Reboot Splunk in order for the new changes to take affect.

Note: Although the above "REGEX" statement may shows on multiple lines it is actually all on one line. Ensure word wrap is off when you paste the text to your transforms.conf file or download the following transforms.conf from here.