Customizing Splunk to parse pfSense logsFor those Basement PC Techs (BPCT) out there that want to send their pfSense traffic to Splunk or have tried and realized that Splunk doesn't automatically parse the logs as it should. Well I got good news for you, I have create the necessary configuration that will allow Splunk to not only parse your data but parses the data the way you want to see your firewall traffic in Splunk by the following fields:
Action (Pass or Block)
The pfSense logs for each firewall event is split into two lines when it is sent to Splunk which Splunk doesn't automatically recognize. By editing two configuration files you can configure Splunk to parse the pfSense event as one so it can be parsed correctly. The two files that we will create/edit are the props.conf and transforms.conf. Each file will need to be created (or edited if one already exist) in the following location:
The props.conf file is were we will configure Splunk to recognized the multi-line events sent from pfSense as one. If you want more detail on what is the purpose of the props.conf file please see the Splunk website: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf
- Create / Edit a props.conf file in $SPLUNK_HOME/etc/system/local/
- Cut and Paste the following into the props.con file:
SHOULD_LINEMERGE = true
TRUNCATE = 0
MUST_NOT_BREAK_AFTER = pf: .* rule ([-\d]+\/\d+)\(.*?\):
MUST_BREAK_AFTER = pf: .* (<|>) +(\d+\.\d+\.\d+\.\d+)\.?(\d*)\:
REPORT-pf2 = pf2
- Save the file
The transforms.conf file is were we will configure Splunk to parse the pfSense events received into the fields we want to see. If you want more detail on what can be done with the transforms.conf file please visit the Splunk website: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf
- Create / Edit a transforms.conf file in $SPLUNK_HOME/etc/system/local/
- Cut and Paste the following into the transforms.conf file:
REGEX= .* (?<action>pass|block) .* (?<protocol>TCP|UDP|IGMP|ICMP) .* (?<src_ip>(\d+\.\d+\.\d+\.\d+))\.?(?<src_port>(\d*)) [<|>] (?<dest_ip>(\d+\.\d+\.\d+\.\d+))\.?(?<dest_port>(\d*)): (.*)
- Save the file
- Reboot Splunk in order for the new changes to take affect.
Note: Although the above "REGEX" statement may shows on multiple lines it is actually all on one line. Ensure word wrap is off when you paste the text to your transforms.conf file or download the following transforms.conf from here.
You rock, this answered my question spot on and got my data into a better format. Just wanted to throw you a thanks!ReplyDelete
I'm not quite sure how the output should be, however, even with the section added to the transforms.conf file I cant see the formatted data. It shows exactly like the raw data with the two merged lines and not the separate fields like I believe it should display. I'm on splunk 5.0.2.
Thanks a lot for your help.
I provide a link to a transforms.conf file that has already been configured to process PFSense logs. Try that file.
I believe that you are asking on how to display the logs better, its done by adding a command like table on your search so that it will display by fields that was extracted by the transforms given above.
index=sample_index|table _time src_ip dest_ip protocol dest_port action
firstname.lastname@example.org for any other questions with field extraction in splunk
Thanks! You should make a Splunk app to make this easier!ReplyDelete
The solution two get *1* line within PFSense and I have this working in pfsense 2.1
Go to Diagnostics -> Edit File, open file /etc/inc/filter.inc and change the following:
From: mwexec_bg("/usr/sbin/tcpdump -s 256 -v -l -n -e -ttt -i pflog0 | logger -t pf -p local0.info"); To: mwexec_bg("/usr/sbin/tcpdump -s 256 -v -l -n -e -ttt -i pflog0 | /usr/bin/sed -l -e 'N;s/\n //;P;D;' | logger -t pf -p local0.info");
You have to reboot your pfsense to see the changes applied, and there no more split lines!
NOTE: credits goes to: http://thwack.solarwinds.com/thread/54381
brillant piece of information, I had come to know about your web-page from my friend hardkik, chennai,i have read atleast 9 posts of yours by now, and let me tell you, your webpage gives the best and the most interesting information. This is just the kind of information that i had been looking for, i'm already your rss reader now and i would regularly watch out for the new posts, once again hats off to you! Thanx a million once again, Regards,splunk training in hyderabadReplyDelete
Cloud Computing Projects
Final Year Projects for CSE
The Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training
Thx for this article, it's nice.ReplyDelete
Wish y all the best.
Toby, data room solutions
From recent time you can use our ProxyInspector for pfSense reporting.ReplyDelete
Here it is an article about how to setup them to work together:
Thanks for giving your time to share this article here about the splunk. Your article is very educational and I am affected by the details that you have shared in this post. Thanks for the wonderful article. splunk Eval CommandReplyDelete
In modern homes, waterproofing the basement is already part of the design process and custom home builders would incorporate all the necessary methods and materials that basements would need to achieve proper waterproofing. Chicago custom remodelingReplyDelete