I had the unfortunate occasion of reconfiguring my OpenWRT router after a fresh reflashing yesterday, without sufficient notes of how I had it set up. So, I'm documenting it this time. But it's not just notes for myself—I'm doing a few tricks that my dear readers might learn from.

First, the topology. I have a /29 subnet: .105–.110 (6 hosts). There is a Cisco DSL modem, a Buffalo router (running OpenWRT), my main server named falcon, a Sipura ATA, a desktop, and my laptop. The latter 4 are collectively the public-IP LAN, and the topology looks like this:

(internet) -- cisco -- buffalo +- public-IP LAN
                               |
                               +- private-IP LAN

The private-IP LAN is 172.17.77.0/24, and primarily for guest wireless devices or whatever other devices I happen to want to plug in.

The buffalo has one internal switch, to which is connected the wireless radio and 5 ethernet ports. It is split into 2 VLANs. vlan1 is the port connected to the cisco (.105), and the other ports and the wireless are bridged as br0 (with a private IP: 77.2).

Now remember, The cisco, buffalo, and public-IP LAN are all on the same /29 subnet. But the cisco and the public-IP LAN are on different VLANs. Let that sink in. You should be objecting about now, and questioning my sanity. (If you're not familiar with the term VLAN, it basically is a way to partition the broadcast domain.) How can I have two independent broadcast segments for one subnet?

Here's what the routing on the buffalo looks like:

# ip route
216.31.27.105 dev vlan1  scope link 
216.31.27.104/29 dev br0  scope link 
172.17.77.0/24 dev br0  proto kernel  scope link  src 172.17.77.2 
default via 216.31.27.105 dev vlan1

and this is how it is achieved:

# cat /etc/init.d/S41routing 
ip route del default
ip route del 216.31.27.104/29 dev vlan1
ip route add 216.31.27.104/29 dev br0
ip route add 216.31.27.105 dev vlan1
ip route add default via 216.31.27.105

So everything in .104/29 goes to the right, and only traffic to the cisco goes to the left. Of course the private subnet goes to the right too. The default route is via the cisco. (Incidentally, you need to install iproute2 for using ip: ipkg install ip)

You also need to configure a static route on the cisco:

set route add ip 216.31.27.104 netmask 255.255.255.248 gw 216.31.27.109

Remember though that the left-hand IP of buffalo is 216.31.27.109, and the right-hand IP is 172.17.77.2. So if falcon, which is on the right, tries to reach .109 (its default route), will it get an ARP response? The answer is, it depends. It depends on the effective arp_ignore sysctl setting. By default on OpenWRT 0.9 (White Russian), /proc/sys/net/ipv4/conf/{default,all}/arp_ignore is set to 1. This means "reply only if the target IP address is local address configured on the incoming interface", which means buffalo won't reply to ARP requests for .109 coming from the right. We need to set arp_ignore to 0 for that interface, meaning "reply for any local target IP address, configured on any interface." How do we do that?

"The max value from conf/{all,$interface}/arp_ignore is used when ARP request is received on the $interface." So, we ned to set conf/{all,br0}/arp_ignore to 0. We do this in /etc/sysctl.conf:

--- etc.orig/sysctl.conf        2007-11-24 08:30:22.000000000 -0700
+++ etc/sysctl.conf     2007-11-24 08:30:29.000000000 -0700
@@ -1,6 +1,7 @@
 kernel.panic=3
 net.ipv4.conf.default.arp_ignore=1
-net.ipv4.conf.all.arp_ignore=1
+net.ipv4.conf.all.arp_ignore=0
+net.ipv4.conf.br0.arp_ignore=0
 net.ipv4.ip_forward=1                 
 net.ipv4.icmp_echo_ignore_broadcasts=1      
 net.ipv4.icmp_ignore_bogus_error_responses=1

There, now (after reboot) the right side of buffalo can ping .109 (don't forget to clear the ARP cache of the host you're testing from), and therefore use .109 as a default route. Everything works great, with one exception. The right side of buffalo cannot ping the cisco. If that bothers you, you can add a static route to the hosts on the right side, like this:

ip route add 216.31.27.105 via 216.31.27.109

But in practice I don't bother. As long as you remember that you won't be able to and don't try to verify routing by pinging the cisco (pick an external IP address instead), you're fine. I can always ssh to buffalo and then have full access to cisco, or just use the serial connection from falcon.

Ok, so that's how I have one VLAN span two broadcast segments. That's not the only thing I do differently, though.

I disable DHCP on buffalo, because falcon is serving DHCP already. But I don't want to disable dnsmasq altogether because the dns bit is still useful, so I edit /etc/dnsmasq.conf and comment out dhcp-authoritative.

I also add this QoS script as /etc/init.d/S42qos.It is the excellent Wondershaper modified for the limitations of the OpenWRT environment. See the comments at the top of the file for instructions. With this QoS setup I don't need to throttle my bittorrent downloads or uploads, but am still able to make clear VOIP calls.

I don't like the default firewall (it's no good for a server behind it), and I find modifying its config is annoying, so I generate a fresh config using ferm and copy it over to buffalo as /etc/firewall.user.

Everything else that I have changed is fairly standard and doesn't need special description.