Installing PFSense on WatchGuard X700

For years I have been running PFSense as my main firewall at home and the equipment that I was using was not what you would say “state of the art”. I have been using an old IBM Pentium 4 Tower which I have never upgraded the system because:

1) it was working and I’m a strong advocate of the old saying “If it’s not broke, don’t fix it”

2) the system allowed me to have 4 nics installed

3) the money that I could use to upgrade the system could be better spent elsewhere (like on my other systems)

The PFSense system that I have ran for years was starting to meet it’s end. The system hardware started to fail and the system would randomly reboot throughout the day which made accessing the Internet challenging so I guess it was time for me to invest in a new system. Originally I thought I would get one of those fanless embedded PFsense devices that start at $199 with only two NICs to replace my old system until a friend at work hip me to using an WatchGuard appliance that could be bought on ebay any where from $25 - $80 with 8+ network connections.  HEY!!! --  I’m all about saving $$ like the next guy and I find a WatchGuard x700 on ebay that had the “Buy Now” option for $59 and free shipping and place the order for the new appliance that would replace my current PFSense system.

I decided to document the steps and any lesson learned from my experience of installing PFSense on my newly acquired WatchGuard appliance. I hope this experience may be helpful to someone else that may be looking for an alternative to an old workstation or a fanless appliance. 
 
Installing PFSense 2.0 on a WatchGuard x700 Appliance.

If you do a Google search of PFSense and WatchGuard, you will get thousand of hits related to the query. There are a few good sites out there that you can use when installing PFSense on a WatchGuard appliance such as the PFSense forums and I must credit the URL at the bottom of this post for actually having the information I required to successfully install PFSense 2.0 on my WatchGuard x700 appliance.

I guess it’s best to list the items you will need while installing PFSense to your WatchGuard appliance. There nothing like having to run to the store to buy something that is needed such as a larger CF Card while in the middle of your PFSense install project. Good thing Walmart is open 24 hours :)



Item List:


1. WatchGuard Appliance
2. CF Card Reader/Writer
3. CF Memory Card 2 to 4 Gigs (recommend 4 GB)
4. Serial Cable
5. PFSense nano image (PFSense Site)
6. Miscellaneous files to upgrade BIOS and post PFSense configuration (see below)

Step 1 -- Remove the CF Card

To access the CF memory card, I had to take out the hard drive tray which I was able to simply pull out of the appliance via the front hard drive access panel and remove the screw to the cover.

Below is a picture of how the inside of my WatchGuard appliance looked after the cover was remove and also the CF Memory Card. Sorry, it wasn’t until I was already into performing my upgrade that I decide to document my experience. In the picture I have highlighted where the CF Memory card would be located. I also pointed out the location of the screws to remove the hard drive tray for easier access to the CF memory card. There were four screws that I had to remove from the hard drive tray holder, two on each side. 

After successfully removing the 64MB CF Card from the WatchGuard, I was able to finally use an accessory that I have bought years ago and hardly ever used....my Digital Concepts 51 in 1 card reader/writer  :)

If you don’t have one, don’t worry. You can pick up one for fairly cheap at one of the following locations:

You don't need a 51 in 1 card just a 1 in 1 that will read and write to your CF Memory Card.

Step 2 - Backup the WatchGuard Firebox CF Image (Optional)

Note: The next steps were all done using a linux system.

Just incase for some strange reason you want to go back to using the factory software that came with the firewall, you can backup the CF Memory card. After connecting the card reader to your computer and inserting the CF Memory Card into the reader, open a terminal window to identify the CF Card connection.

Type: “sudo fdisk -l” 

(This will list all drives connected to your system. Look for the disk that is the size of your CF Card. Mines was 64 MB and the output of the fdisk command on my system is listed below:

sudo fdisk -l

Disk /dev/sdd: 64 MB, 64225280 bytes
1 heads, 62 sectors/track, 2023 cylinders
Units = cylinders of 62 * 512 = 31744 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

  Device Boot   Start      End   Blocks   Id  System
/dev/sdd1           34      302     8320   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sdd2          302     1128    25600   83  Linux
Partition 2 does not end on cylinder boundary.
/dev/sdd3         1128     1954    25600   83  Linux
Partition 3 does not end on cylinder boundary.
/dev/sdd4         1954     1987     1024   83  Linux
Partition 4 does not end on cylinder boundary.
According to the output of the fdisk command, the card was located at “/dev/sdd” which contained 4 partitions listed as /dev/ssd1-4.

Now that I have confirmed the device location of the CF memory card “/dev/ssd” I’m now ready to make a backup of the card. Using the “dd” command I created a backup to file

Command Syntax: sudo dd if=/disk device location  of=/location and name to save backup image

Command used: “sudo dd if=/dev/sdd of=WatchGuardBackup.img”

Note: of=WatchGuardBackup.img will save the backup image to a file called WatchGuardBackup.img in the current location of the terminal window.

Step 3 - Flash the BIOS of the WatchGuard Appliance

Note: The next steps were all completed using a Linux system.

The next step in getting PFSense to run on a WatchGuard appliance is to flash the BIOS to enable support for larger CF Memory Cards. The WatchGuard appliance currently will not boot from a CF card larger than 256MB and by flashing the BIOS with the X750EB6.BIN file, your WatchGuard appliance will now be able to support CF Memory Cards larger then 256MB.

1. Download the FreeDOSBios.img and X750EB2.BIN BIOS image
2. Write FreeDOSBios.img image to CF Memory Card.

Using the location identified in “Step 2 - Backup the WatchGuard Firebox CF Image” I proceeded to write the FreeDOSBios.img to the CF Memory card. From the terminal window, I ran the following command:

Command Syntax: sudo dd if=/path to FreeDOSBios.img/ of=/disk device location/

Command used: “sudo dd if=/Download/FreeDOSBios.img of=/dev/sdd”

3. After writing the FreeDOSBios.img to your CF Memory Card, mount the CF memory card and copy the X750EB2.BIN file to the bios folder on the CF Card. After the file has successfully been copy to the “bios” folder, you can unmount the CF Card and install it back into the WatchGuard Appliance.

4. Connect your serial cable from your computer to the WatchGuard appliance. Once again, I get to use something that’s has been sitting in a box in my closet for years, my USB to Serial converter. With more than 10+ computers in my household, not one besides my old IBM PFSense system has a serial port. I believe serial ports are a things of the past. When I take my current PFSense system apart I have to make sure I save the serial card. I will place in the box next to the 5 ¼ and 3 ½ disk I have. You never know when you may need some ancient technology in the future LOL.

Note: If you too also do not have a computer with a serial port, you can get a USB to Serial adapter from one of the following sites:



After connecting your serial cable to your computer and the WatchGuard appliance, you want to start a terminal session on the serial cable. To ensure what device your serial cable is using on your system you can check the dmesg log.

In a terminal window type the "dmesg" command. This should display the dmesg log file.   

Below is a copy of the tail of my dmesg log file:

[ 2960.116026] usb 5-2: reset full speed USB device using uhci_hcd and address 3
[ 2960.236021] usb 5-2: device descriptor read/64, error -71
[ 2960.460021] usb 5-2: device descriptor read/64, error -71
[ 2960.676021] usb 5-2: reset full speed USB device using uhci_hcd and address 3
[ 2962.112029] usb 5-2: reset full speed USB device using uhci_hcd and address 3
[ 2962.232020] usb 5-2: device descriptor read/64, error -71
[ 2962.456014] usb 5-2: device descriptor read/64, error -71
[ 2962.672021] usb 5-2: reset full speed USB device using uhci_hcd and address 3
[ 3990.232024] usb 5-1: new full speed USB device using uhci_hcd and address 4
[ 3990.505878] usbcore: registered new interface driver usbserial
[ 3990.505889] USB Serial support registered for generic
[ 3990.505915] usbcore: registered new interface driver usbserial_generic
[ 3990.505917] usbserial: USB Serial Driver core
[ 3990.514510] USB Serial support registered for pl2303
[ 3990.514532] pl2303 5-1:1.0: pl2303 converter detected
[ 3990.525896] usb 5-1: pl2303 converter now attached to ttyUSB0
[ 3990.525916] usbcore: registered new interface driver pl2303
[ 3990.525919] pl2303: Prolific PL2303 USB to serial adaptor driver

On my system my the serial cable is using ttyUSB0. If you are unsure which device your serial cable may be using, disconnect the cable from your system view the dmesg log again to see if your can identify the cable being disconnected and then connect it back to your computer again while reviewing the dmesg log again.

Once you know what device your serial cable is using, open a session to your serial cable by using the "screen" command. Typing the following command then turn on the WatchGuard appliance:

Command syntax: screen /dev/{your serial cable device} 9600


Command used: “screen /dev/ttyUSB 9600”

Once you enter the "screen" command in the terminal window, your window will go blank until the WatchGuard appliance fully loaded. My appliance sounded 3 beeps when device had finished booting up and I was presented with the following:

Freedos on COM1:
Current date is Mon 11-05-2012
Current time is  3:44:40.80 am
C:\>

Once you are presented with the “C:\>” prompt, you are now ready to flash the BIOS with X750EB6.BIN file.

C:\>cd bios
C:\BIOS>awdflash.exe X750EB6.BIN /py /sn /cc /e

Lesson Learned Time: Being a person very comfortable with using command line, there were some command line features unavailable in the "screen" terminal session to the serial console. Two such features that have caused me to restart this step many times was the missing tab to autocomplete your commands feature and the inability to use the backspace key. Neither was available while using the "screen" command to access the serial cable. Trying both features sent unrecognizable characters to the terminal window which made my session unresponsive. I was able to regain responsivness after shutdown the WatchGuard appliance.

After you have successfully flashed the BIOS, you can power off the WatchGuard appliance and remove the CF card.

Step 4 -- Installing PFSense on your CF Memory Card

Now it’s time to install PFSense on your CF memory card.

1. Download the PFSense nanobsd Image image for the size of your CF memory card. I downloaded the  pfSense-2.0.1-RELEASE-4g-i386-nanobsd.img.gz to install on my 4 GB card.
2. Decompress the .gz file via the "tar" command:

Command Used: tar -xvf ./pfSense-2.0.1-RELEASE-4g-i386-nanobsd.img.gz
 
3. Write the PFSense image to your CF Memory card. Connect your CF memory card to your card writer and identify the disk device as previously done in “Step 2 - Backup the WatchGuard Firebox CF Image”. Write the PFSense image you downloaded and decompressed to the CF Memory card by using the following "dd" command:

Command Syntax: dd if=/{path to PFSense image}  of=/dev/{disk device} bs=16k

Command used: sudo dd if=./pfSense-2.0.1-RELEASE-4g-i386-nanobsd.img of=/dev/sdd bs=16k

4. Install CF Memory Card with PFSense installed in the WatchGuard appliance and open a session with your serial cable as you did previously using the "screen" command:

Command Syntax: screen /dev/{your serial cable device} 9600

5. Power on your WatchGuard Appliance and this time you should be presented with the PFSense initial setup Wizard.

From this point on you would set up PFSense just like a regular installation.

See How to Install PFSense and How to Configure PFSense.


I want to thank the following sites with helping me to successfully install PFSense on my Watchguard appliance.

http://www.nettechonline.net/index.php/pfsense/pfsense-watchguard/78-x700-led-fix-pfsense-on-watchguard.html?catid=49%3Apfsense-watchguard

http://www.nettechonline.net/index.php/pfsense/75-x700-a-pfsense-20-rc1.html?catid=49%3Apfsense-watchguard

http://practicalkungfu.net/2012/02/20/how-to-install-pfsense-2-0-on-a-watchguard-x750e-core/

Malware Memory Analysis - Volatility

In the Acquiring Memory blog a list of tools that could be used to acquire the memory of a live system was listed. Once you have successfully acquire the memory of the system, a tool like volatility can be used to analyze the memory for data. In this assessment I will be evaluating the memory sample as a person that has no formal training in memory analysis or on how to use the tool to see if I can still use the tool to identify malicious code contain within the memory image. For this test the Zeus memory sample acquired from the Google Code – Volatility Memory Sample page will be used.

I will use practical troubleshooting steps to established my approach of analyzing the memory sample.

• Look for strange processes
• Look for strange network connections
• Check registry for strange entries added by the malicious code.
• Analyze suspicious code

Volatility

Here is the official description of the tool from the developer page:

“The Volatility Framework is a completely open collection of tools, implemented in Python under the GNU General Public License, for the extraction of digital artifacts from volatile memory (RAM) samples. The extraction techniques are performed completely independent of the system being investigated but offer unprecedented visibility into the runtime state of the system. The framework is intended to introduce people to the techniques and complexities associated with extracting digital artifacts from volatile memory samples and provide a platform for further work into this exciting area of research. “

http://code.google.com/p/volatility/

Installation

Installation of volatility is very simple. If you are using Windows, it just a matter of downloading the version you want and running the installer. For UNIX base systems, the hardest part of the installation may be determining where you want to extract the the source code after you download it. Once the source code has been extracted, your ready to use volatility. No compiling needed.
Note: There is one main requirement when it comes to using Volatility – python. You must have a working version of python installed on your system. If you are using an up to date UNIX base system to include OS X, you already have a working version of python installed on your system. Windows users may want to download and install the volatility standalone version. It comes with everything you need to use volatility including a runtime version of python. Check the download page on the volatility main website for more detail.

Using Volatility

Basic usage

Basic usage of volatility is as follows:


$ python vol.py [plugin] -f [image] –profile=[PROFILE]

The options after the “vol.py” command can be in any order as long as all like options are together. During my evaluation, I discovered that it was best to use the following order when running volatility from the command line. It allowed me to edit the last command ran from the command line easier.

$ python vol.py –profile=[PROFILE] -f [image] [plugin]

 

Volatility Commands

During this test I essentially reference the volatility command reference page which listed all the commands along with sample output of each commands. All similar commands are group together under the below section on the CommandReference page :

• Image Identification
  
• Processes and DLLs
  
• Process Memory
  
• Kernal Memory and Objects
  
• Networking
  
• Registry
  
• Crash Dumps, Hibernation, and Conversion
  
• Malware and Rootkits
  
• Miscellaneous
  

Command Reference Page: http://code.google.com/p/volatility/wiki/CommandReference
 

Step 1 - Image Identification

Having not capture the memory sample myself, it seem logical to run the plugin command “imageinfo” against the Zeus memory image to see what information the plugin can provide about the memory image.

python vol.py imageinfo -f zeus.vmem

The “imageinfo” plugin provided some valuable information such as the date and time of the image and local date and time of the system the memory was acquired. It also suggested which profile should be used when using the “--profile” option with volatility. The WinXPSP2x86 profile was the profile suggested to be used for this Zeus memory sample.

Step 2 - Looking for strange processes

Using the CommandReference page, I identify the plugin command to list the processes that were running in the memory sample when the memory was acquired.

python vol.py --profile=WinXPSP2x86 pslist -f zeus.vmem

At 1^st glance, there appears to be no strange processes running.

Step 3 - Looking for Strange Network Connections

The next step in my hunt for malware was to check the memory for any strange connections. Once again I used the CommandReference page to identify the plugin command to use to list all current network connections:

python vol.py –profile=WinXPSP2x86 connections -f zeus.vmem

Just my luck, no current connections were found. Well right under the “connection” plugin command on the CommandReference page was the command to scan for previous connections found in the memory - “connscan”. I tried that plugin command next.

python vol.py --profile=WinXPSP2x86 connscan -f zeus.vmem

Ahhh Yea, two connections found and now I just to learn more about the Remote Address. Using freegeoip.net website I enter the remote IP Address to learn the geolocaton of the IP.

Freegeoip.net http://freegeoip.net/static/index.html

Yes I agree that I could have just used ARIN to do a simple whois on the remote address, but I'm a visual person and I tell you that I love seeing that little green arrow pointing the possible geolocation of the Remote Address which in this case is in the Replublic of Moldova. It's so much cooler to look at then looking at the following:

Now we know where the Remote Address is located but this is not the only valuable piece of data we can acquire from “connscan” output. The “connscan” output also displays the process ID (PID) of the process that was associated with the connection to the Remote Address which happens to be PID 856.

Referring back to the output we got from the “pslist” plugin, we identify that “scvhost.exe” is the process that was associated with the connection to the Remote Address. According to the “plist” output we also see that the parent process ID (PPID) was 676 (services.exe) which in turn was started by PID 632 (winlogon.exe)

PID 632 winlogon.exe

       PID 676 services.exe

              PID 856 (svchost.exe)

                     Remote address: 193.104.41.75

Step 4 - Dump process

So from my previous outputs I have identified at least two processes that I want to take a closer look at “svchost.exe” and the process that started it “services.exe”. Using the volatility plugin command for dumping a process from the memory image - “procexedump” I proceeded to dump each process.

python vol.py --profile=WinXPSP2x86 -f ../zeus/zeus.vmem -p 856 procexedump -D ../zeus/

python vol.py --profile=WinXPSP2x86 -f ../zeus/zeus.vmem -p 676 procexedump -D ../zeus/

I submitted both processes that was dump by volatility to VirusTotals.com for analysis and received the following results:

svchost.exe analysis

The “svchost.exe” process had 1 detection by Emsisoft for Packed.Win32.Krap.g!A2 out of the 42 antivirus scanners that scanned the process.

Services.exe analysis

The “service.exe” process had no detection out of the 42 antivirus scanners that scanned the process.

Volatility also offers an additional plugin for dumping the process and it's slack space by using the plugin command “procmemdump”. I decided to dump the processes again using this option to see if my results would change.

Note: unless you specify a different output location or rename any previous PID process output, volatility will overwrite any process dumps with the same PID. I found this out the hard way LOL

Dumping the processes including slack space from memory

python vol.py --profile=WinXPSP2x86 -f ../zeus/zeus.vmem -p 856 procmemdump -u -D ../zeus/

python vol.py --profile=WinXPSP2x86 -f ../zeus/zeus.vmem -p 676 procmemdump -u -D ../zeus/

As before, I submitted the dumped processes to VirusTotal.com and this time the results were more interesting.

svchost.exe with slack space analysis

The “svchost.exe” process had 6 detection out of the 42 antivirus scanners that scanned the process.

Antivirus	Results
AntiVir	TR/Crypt.XPACK.Gen
CAT-QuickHeal	(Suspicious) – DNAScan
Comodo	UnclassifiedMalware
Emsisoft	Trojan.Crypt!IK
Esafe	Win32.TRCrypt.XPACK
Ikarus	Trojan.Crypt

Services.exe with slack space analysis

The “service.exe” process had 2 detection out of the 42 antivirus scanners that scanned the process.

AntiVirus	Results
CAT-QuickHeal	(Suspicious) – DNAScan
Symantec	WS.Reputation.1

So it appears that if you are going dump a process out of memory that it's best to use the “procmemdump” plugin as compare to “procexedump” plugin. Now I'm pretty satisfied that I was able to find some malicious code in the memory image but I don't really know how it got there. Maybe the registry can provide me with that answers.

STEP 5 – Search the Registry for signs of Malware

Now it time to search the registry for suspicious entries. I know of the normal places to search the in the registry for signs of malware such as “Run” keys but I decided to do a search on the Internet to see if there may actually be a list of the most used registry keys by malware and not to be disappointed, there was a list of the top 10 registry keys to look for signs of malware by F-Secure.

Top 10 Launch points

http://www.f-secure.com/weblog/archives/00001207.html

With list in hand now it time to use volatility to search the registry hives contained within the memory. Referring back to the Volatility CommandReference page, I identify the plugin command that would list the current registry hives in memory - “hivelist”. I used this plugin command to list the registry hives in my memory sample:

python vol.py --profile=WinXPSP2x86 -f ../zeus/zeus.vmem hivelist

From the output of the “hivelist” plugin, I see that the following registry hives are available:

• Software
• SAM
• System
• NTUSER.DAT

This is great because the hives that I need to review according to the F-Secure top 10 list are present.

Displaying values in the registry

I review all the registry key as suggested by the F-Secure list using the “printkey” plugin in the following convention:

python vol.py --profile=WinXPSP2x86 -f ../zeus/zeus.vmem printkey -o {virtual address} -K {regestry key}

• {virtual address} is the address acquired from the “hivelist” plugin.
• {registry key} is the path of the registry key you want to display the data value

After searching all the keys in the F-Secure top 10 list, only one actually had something suspicious. The “Microsoft\Windows NT\CurrentVersion\Winlogon” key.

python vol.py --profile=WinXPSP2x86 -f ../zeus/zeus.vmem printkey -o 0xe153ab60 -K "Microsoft\Windows NT\CurrentVersion\Winlogon"

The “Userinit” key seems to have an additional value added which will start at login - “sdra64.exe”. Since we do not have the actual system at hand that the memory sample was acquired to upload the sdra64.exe file to VirusTotal.com to be examine I performed a Google search on “sdra64.exe” to see what I could find. The Google Search returned over 56,400 results in just .25 seconds which many referred to the files as being the Zbot. I think I found the origin of the Zeus infection.

Malware Memory Analysis

Acquiring RAM From A Live System

In the not so far past, it was common practice when encountering a system that needed to be acquired or has been compromised to disconnect the power from the system in order to save the "state" of the system to be forensically analyzed later. We have since learned that valuable data is lost when the system is powered off and all attempts to collect this volatile data should be capture before the system is powered off. This may include but not limited to the following:

• current network connections
• running processes
• current mapped drives or shares
• Users currently logon
• system memory

Acquiring the memory from a system is a relatively new item to acquire when it comes to collecting volatile data. It wasn't until the recent years that we learn how to analyze this data in order to extract the valuable data contain within the memory.

Some of this valuable data that can be found in the memory consist of the following:

• Current process and DLLs
• Network connections
• Unencrypted password
• Registry entries

There are many tools available that you can use to acquire the live memory from a system. Below is a list of some of the popular free tools:

Tool	OS	Comments
FTK Imager	Windows	Lite version does not need to be installed and can run from usb
HBGary FastDump	Windows	Support only 32bit up to 4 gigs of RAM Does not support Vista, Windows 2003, Windows 2008
MoonSols DumpIt	Windows	works with both x86 (32-bits) and x64 (64-bits) machines
Mandiant Memoryze	Windows	Officially Supports: Windows 2000 Service Pack 4 (32-bit) Windows XP Service Pack 2 and Service Pack 3 (32-bit) Windows Vista Service Pack 1 and Service Pack 2 (32-bit) Windows 2003 Service Pack 2 (32-bit) Windows 2003 Service Pack 2 (64-bit) Windows 7 Service Pack 0 (32-bit) Windows 7 Service Pack 0 (64-bit) *Windows 2008 Service Pack 1 and Service Pack 2 (32-bit) Windows 2008 R2 Service Pack 0 (64-bit)
dd	*nix	Comes standard on most *nix systems can be used to capture the contents of physical memory using a device file (e.g. /dev/mem and /dev/kmem)
Mac Memory Reader	Mac	It executes directly on 32- and 64-bit target machines running Mac OS X 10.4 through 10.7 and requires a PowerPC G4 or newer or any Intel processor.

Once you acquire the system memory, then comes the task of extracting the data from the image. Below are three tools that can be used to extract this data and over the next couple of blog postings, I will use each tool on the same memory samples to evaluate the ease of use of each program along with their differences. I will utilize and evaluate each program as a person that has not had any formal training on using each tool or malware analysis but can deploy common troubleshooting skills to identify the threats contained within each memory sample. All samples used for this evaluation will be acquired from the publicly available sample found on the Google Code - Volatility page.

http://code.google.com/p/volatility/wiki/MemorySamples

The three applications used for memory analysis are:

• Volatility
• Mandiant Redline
• HBGray Responder (community edition)