Network resilience
Each server is connected to two different Ethernet switches for resilience – a switch failure is thus a non emergency situation.
Warning: Configuring bonding may not work first time, confirm that you have a working out-of-band connection (serial / LOM) to your server before trying to configure bonding so that you can recover the system if you’re left without normal network access to it.
To configure on Ubuntu 11.10
We need the ifenslave package installing
# apt-get install ifenslave
The bond module needs loading at boot
# echo "bonding" >> /etc/modules
Edit /etc/network/interfaces
auto eth0 iface eth0 inet manual bond-master bond0 bond-primary eth0 eth1 auto eth1 iface eth1 inet manual bond-master bond0 bond-primary eth0 eth1 auto bond0 iface bond0 inet static slaves eth0 eth1 bond-mode 6 bond-miimon 100 address ..... netmask etc as per original eth0
Check the bonding is working
# cat /proc/net/bonding/bond0
More information
The switches are independent of each other, and consider the two links to be separate connections, it is important that any particular mac address is not regularly sourced from both of the links as this will cause mac table churn on the switches.
The functionality in operating systems for considering two links to be part of the same interface is often geared to bandwidth aggregation rather than resilience, so care needs to be taken to ensure that both links aren’t sourcing the same mac address regularly.
More information (linux specific)
The linux bonding driver is documented at http://www.kernel.org/doc/Documentation/networking/bonding.txt
https://help.ubuntu.com/community/LinkAggregation has details of how to do network resilience on Ubuntu. Note that the driver has a primary intended use of bandwidth increase, not resilience
Do not use bond mode 0 (balance-rr), 2 (balance-xor), or 3 (broadcast) as these source frames with the same mac address on both ports.
Modes 1 (active-backup), 4 (802.3ad), 5 (balance-tlb), 6 (balance-alb) are potentially acceptable, of these 4 is pointless (we can’t run ports on two different switches as part of an 802.3ad bundle, also in testing fail-back leads to other failures), and 1, 5, and 6 offers simplicity, transmit load balancing, and transmit and receive load balancing respectively.