Never Restart Network on Multi-ADSL Connection Machine…

I learned it the hard way. Our gateway machine with firewall (shorewall) has multiple ADSL connections configured with load balancing for more bandwidth and transparent fail-over. Today I faced an unenviable problem where one or other of the ADSL connections were going down sometime after a network restart. It was unique because the ADSL modems (configured as router) were accessible via telnet or ping and displayed perfect connectivity. However I was unable to use them as gateway to connect to a server on the internet. They were working fine before. With our non-trivial setup there were many suspects including the shorewall firewall and iptables (dropping certain packets?), network adapter, routing issue etc.

The solution was interesting. We need to make some changes to the routing table to allow simultaneous access through multiple network adapters. In our case they were these:

ip route add 192.168.1.0/24 dev eth1 src 192.168.1.10 table bsnl
ip route add default via 192.168.1.1 table bsnl
ip route add 192.168.0.0/24 dev eth2 src 192.168.0.10 table tata
ip route add default via 192.168.0.1 table tata
ip rule add from 192.168.1.10 table bsnl
ip rule add from 192.168.0.10 table tata
ip route replace default scope global nexthop via 192.168.1.1 dev eth1 weight 1 nexthop via 192.168.0.1 dev eth2 weight 4

When the network is restarted all these entries are lost. So when eth2 comes up later, eth1 goes down and vice versa. The solution is to use a script to restart the network which does the following:
1. Restart network
2. Add entries to routing table similar to example above
3. Restart shorewall firewall (required after network restart)

Now I can safely restart the network without any issues at all.