Steve T's Semi-Cohreant Ramblings

The goal was to put a client end box behind some sort of router/gateway device
that would introduce latency and if possible bandwidth restrictions and random
packet loss.

Research indicated the best option is to use dummynet. Dummynet is a flexible
tool originally designed for testing network protocols. It can also be used
for bandwidth management (although the owner claims it is misused).

It is a built in feature of FreeBSD so that is the route I took.

The first step is to download FreeBSD which is now version 7. I download the
torrent file and used the linux command line client “rtorrent” to pull down
the ISOs. I only really needed Disk 1.

The server I found to do this task is a typical older server most IT shops will
have laying around. It is a Dell 1750 with a 2.8GHz processor and 1G of RAM.
That will be plenty of horsepower to do the routing required.

Boot the new router with FreeBSD Disk 1 and begin the install. When it asked about the partitions I used a single FreeBSD partition (type 165) made of the whole disk. For the boot manager I selected the FreeBSD BootMgr. When it came to file system layout, I used the “auto defaults”. It created enough swap and user space for me (even on a 20 Gb drive). When I selected which version to install I used the “developer + ports” version. I made sure
not to install any GUI features (X, Gnome, and the like) because this will be a
headless server sitting in a data center. Give the primary NIC an address on
your LAN and a reasonable name. I named my server “cloud” because it will
represent the “Internet Cloud” when it is fully functional. The first time I did the install, I did not do the developer version which includes all of the source tree. When it is time to update the system it will take a lot longer to update if it is not put down on the initial update. Also, be sure to install the ports tree. It will prove to be really helpful.

Once your system is up and running it is time to install some basic packages
and update the source tree.

I installed the following packages: bash, cvsup-without-gui, and
isc-dhcp3-server. One of the wonderful things about FreeBSD is the ports
system. From within /usr/ports you have access to over 18000 third party
packages with all dependencies already worked out. To install bash for example
you use the following commands:
# cd /usr/ports
# cd shells/bash
# make install clean

That’s it. Simple. I changed root’s default shell to bash because I am more
comfortable working with bash (chsh -s /usr/local/bin/bash). The other two
packages I used are found in /usr/ports/net/cvsup-without-gui and
/usr/ports/net/isc-dhcp3-server.

The next step is to compile a new kernel and world. This is required because
we need to make dummynet a known option of the kernel you are running. Before
we even get close to configuring the kernel, it is a good idea to update the
source tree. That is why we installed cvsup above.

When cvsup was installed a couple of example cvsup config files were put on
the system is /usr/share/example/cvsup. The one we are interested in is
“stable-supfile”. Copy that file to a working name such as “active-stable-supfile”. That way it will not get overwritten with the updates. Edit the new file and change the host=CHANGE_THIS.FreeBSD.org to a real cvsup host. Here is a list of cvsup mirrors. I do recomend not using the Central Servers and using one of the Primary Mirrors. Then you just need to run cvsup.
# cvsup -g -L 2 active-stable-supfile

This will probably take quite some time. If you make the mistake I made of not installing the source initially, this will take 2 or 3 hours depending on your network connection. When the cvsup is complete it is time to begin the buildworld process. It is a simple command but will take quite some time.
# cd /usr/src
# make buildworld

When that is complete you then need to build your custom kernel. Why is it
custom? Because we need to add a few options to the kernel configuration
file. To make this config file copy the generic config file. I used CLOUD because that is my machine name.
# cd /usr/src/sys/i386/conf
# cp GENERIC CLOUD


Edit the “CLOUD” file and add these three options. I put them at the end of
the options section but above the devices section:
options IPFIREWALL
options DUMMYNET
options HZ=1000

Now you are ready to compile your kernel. It is also pretty simple:
# cd /usr/src
# make buildkernel KERNCONF=CLOUD


When that is complete, it is time to install your new custom kernel. Simple:
# make installkernel KERNCONF=CLOUD

Now you need to reboot into single user mode. I have found the easiest way
to do this is to just issue the “reboot” command and sit at console. When the
FreeBSD boot menu appears select option “4” which is boot into single user mode.

When you are in single user mode you will want to do the following commands:
# mergemaster -p
# make installworld
# mergemaster
# reboot

It sounds simple, but mergemaster can be a bit of a pain. Since this is a
brand new system you can almost always choose “i” to install the temporary
version of the file. If you had done any custom work prior to this merge
you will need to make sure your changes do not get undone.

When you boot to full multi-user and login run this:
# uname -a
FreeBSD cloud.qa.example.com 7.0-STABLE FreeBSD 7.0-STABLE #2: Tue Jun 17 17:17:47 CDT 2008 [email protected]:/usr/obj/usr/src/sys/CLOUD i386

Now you are ready to make the router. FreeBSD really shines here because you
just need to make a couple of config changes.

First, set up your second network adapter. On my 1750 this is bge1. In
/etc/rc.conf I added these three lines:
ifconfig_bge1="inet 10.250.15.1 netmask 255.255.255.0"
gateway_enable="YES"
ipnat_enable="YES"

Then I created a file called “ipnat.rules” with these two lines:
map bge0 10.250.15.0/24 -> 10.10.10.5/32 portmap tcp/udp 40000:65000
map bge0 10.250.15.0/24 -> 10.10.10.5/32

The “10.10.10.5” address is my network address to the outside world, adjust
that as necessary.

Now you need to setup DNS forwarding. Named is already installed on your
system you just need to enable and configure it. In /etc/rc.conf add this
line:
named_enable="YES"

Then edit /etc/namedb/named.conf. Comment out the “listen-on” entry so it
looks like this:
// listen-on { 127.0.0.1; };

That will enable named to listed for anything on the network. Then un-comment
the “forward only” line. You also need to un-comment the forwarders section
and enter your upstream DNS server. Mine looks like:
forwarders {
10.10.10.71;
};

To make it easier to have clients on your new internal network you may want to
configure the dhcpd server installed above. The file is located at
/usr/local/etc/dhcpd.conf. Mine looks like this:
option domain-name "example.com";
option domain-name-servers 10.250.15.1;
option subnet-mask 255.255.255.0;
default-lease-time 86400;
max-lease-time 86400;
ddns-update-style none;
subnet 10.250.15.0 netmask 255.255.255.0 {
range 10.250.15.150 10.250.15.200;
option routers 10.250.15.1;
}


You also need to enable the dhcpd server by adding this line to your rc.conf
file:
dhcpd_enable="YES"

There a few other options in rc.conf you should add now because we will need
them for dummynet specifically:
firewall_enable="YES"
firewall_type="OPEN"
dummynet_enable="YES"

Now make sure your 2nd NIC is plugged into a hub/switch/vlan segment and reboot.

When your system comes up an ifconfig should reveal that you have both NICs up
and the network configured. The command “ipnat list” should show your two
NAT rules from above.

Hook up a client box to the internal network segment. Use DHCP and get a
lease. When you have a lease, do a nslookup of a known host (like www.cnn.com).
Fire up a web browser and make sure you can surf the web. Life should be good
at this point.

Now we are at the point of this computing adventure. Time to add latency!

On your client box set up a continuous ping. For windows this would look like:
C:> ping -t www.google.com
Pinging www.l.google.com [74.125.47.147] with 32 bytes of data:
Reply from 74.125.47.147: bytes=32 time=36ms TTL=241
....
Reply from 74.125.47.147: bytes=32 time=37ms TTL=241

Now on your new network device issue these two commands:
# ipfw pipe 10 config delay 50
# ipfw add 1000 pipe 10 all from 10.250.15.0/24 to any

Look back at your client box. The ping times should jump 100ms. The 50 from
the first command is actually done twice, once on the way out and once on the
way back in. You have just added 100ms latency to your connection.

Great, now it is time to remove that latency. These two commands will do it:
# ipfw delete 1000
# ipfw delete pipe 10

Poof. Latency gone.

You now have a router with a high cool factor. Why cool? Because you have a lot more control over your internal network than your normal plug-n-play routers.