GIGABIT CAMPUS NETWORK DESIGN 1
Content
W HITE P APER
Gigabit Campus Design
Configuration and Recovery Analysis
Introduction
Some of the most useful variations of the large-scale multilayer campus internetwork design were tested for failure recovery. Please refer to the companion document Gigabit Campus Design—Principles and Architecture for an explanation of several implementations of the multilayer design. In all cases, Gigabit EtherChannel® links were used between distribution-layer switches and core-layer switches. Redundant Gigabit Ethernet uplinks were used to connect the access-layer switches to the distribution-layer switches. The Catalyst® 4000, 5500, and 6000 family switches were used in the wiring closet configurations. The Catalyst 6500 with Multilayer Switch Feature Card (MSFC) and Catalyst 8540 were both used as the Layer 3 switch. The Catalyst 6500 is used as the Layer 2 switch in the core. This paper documents configuration and recovery with three fundamental designs: 1. Layer 3 distribution with dual-path Layer 2 core (good) a) Catalyst 6500 standard building block 2. Layer 3 distribution with Layer 3 core (better) a) Catalyst 6500 standard building block 3. Layer 3 distribution with dual-path Layer 3 core (best) a) Catalyst 6500 standard building block b) Catalyst 6500 virtual LAN(VLAN) building block The following two design variations are described in the appendix: A1. Layer 3 distribution with Layer 3 core a) Catalyst 8540 standard building block A2. Layer 3 distribution with dual-path Layer 3 core a) Catalyst 8540 standard building block Layer 2 spanning-tree loops were eliminated in the core and in the standard building block. Avoidance of spanning-tree loops in the core is important for high availability. Within the core and the standard building block all Layer 3 switches were configured with native routed interfaces. VLAN trunks were only used within the VLAN building block. The VLAN building block is typically used in a server farm to provide for redundant server connection with dual Network Interface Cards (NICs). The Enhanced Interior Gateway Routing Protocol (EIGRP) was tuned for fast convergence with a one-second-hello timer and a three-second-hold timer. The Hot Standby Routing Protocol (HSRP) was tuned for fast convergence with a one-second-hello timer and a three-second-hold timer. When using protocol hello timers of one second, the limit of accuracy for recovery measurement is about one second.
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 1 of 26
Thus, all results are rounded to units of whole seconds, representing the worst case. Network recovery times were very predictable, with some variation due to the state of things such as the Address Resolution Protocol (ARP) cache. OSPF was configured with the same timer values as EIGRP, with similar results. Every router and switch in the network was provided an out-of-band Ethernet management interface on VLAN 99. One Ethernet port on each device was configured in VLAN 99 and wired outside the test network to a separate switched management network. In addition to the management IP address on VLAN 99, we also configured Cisco 2511 terminal servers with RJ-45 serial cables to access the console port of every router and switch in the layout. With this arrangement, we have two out-of-band paths to every switch and router, as well as a number of in-band paths. Wherever possible Gigabit EtherChannel was configured with ports on two different cards to increase availability. For example, port-channel 1 on CT85 (core top 8540) used interfaces gigabit 0/0/0 and gigabit 1/0/0, which are the first physical ports on two different cards. EtherChannel recovery was not tested, as it is much faster than one second and falls below the measurement threshold of this testing.
Configuration of the Standard Building Block
The standard building block appears on the left side (west block) of Figures 3a, 3b, and 3c. Configuration of the standard building block is very simple because all spanning-tree loops are eliminated. Hence, there is no Layer 2 tuning, such as selecting the best root switch for a given VLAN. No VLAN trunks are used and uplinks are connected to native routed interfaces on the Layer 3 switches in the distribution layer.
Catalyst 6000 Access Switch: Standard Building Block
Set prompt aw6 (access layer, west block, Catalyst 6000) Set vtp domain west Set vtp mode transparent (no VLAN trunks, use transparent mode) Set vlan 99 (used for out of band management) Set vlan 99 3/48 (last physical port on switch used for oob management) Set int sc0 99 172.29.196.51 255.255.254.0 (logical console port for oob management) Set ip route default 172.29.196.1 (gateway router in oob management network) Set port channel 1/1-2 mode off (turn off channel negotiation on uplinks) (not using EtherChannel in this configuration) set trunk 1/1-2 off(turn off VLAN trunking on uplinks) set vlan 10(VLAN 10 corresponds to subnet “10” ie 10.10.0.0) set vlan 10 1/1-2(all other ports are part of VLAN 10) set vlan 10 3/1-47
We disable channel negotiation on the uplinks to make connection faster after a failure has been restored. This is appropriate because the uplinks are routed connections to Layer 3 switches in the distribution layer, not Layer 2 connections that require spanning tree. (Nothing in the configuration indicates that all clients and uplinks attached to the switch are in subnet “10” (10.10.0.0) with mask 255.255.0.0. VLAN 10 is configured everywhere to match subnet “10” [10.10.0.0]).
Configuration of the Standard Building Block with Load Balancing
To achieve load balancing with the standard building block, a couple of changes are required. Configure two VLANs (two subnets) on a wiring closet switch and use VLAN trunks for uplinks. For example, on switch aw6, configure VLAN 10 and VLAN 11. All outbound VLAN 10 traffic will take one uplink to the HSRP primary gateway router for subnet 10.10.0.0. All outbound VLAN 11 traffic will take the other uplink to the HSRP primary gateway router for subnet 10.11.0.0. This load balancing configuration maintains all the advantages of the standard building block and is only marginally more complex.
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 2 of 26
Catalyst 6000 Access Switch: Standard Building Block with Load Balancing
Set prompt aw6 (access layer, west block, Catalyst 6000) Set vtp domain west Set vtp mode transparent (use transparent mode, configure VLANs explicitly) Set vlan 99 (used for out of band management) Set vlan 99 5/48 (last physical port on switch used for oob management) Set int sc0 99 172.29.196.51 255.255.254.0 (logical console port for oob management) Set ip route default 172.29.196.1 (gateway router in oob management network) Set port channel 1/1-2 mode off (turn off channel negotiation on uplinks) (not using EtherChannel on this switch) set trunk 1/1-2 on 10,11 dot1q (VLAN trunking on uplinks, for VLAN 10 and VLAN 11) set vlan 10 3/1-48(VLAN 10 corresponds to subnet “10” ie 10.10.0.0) set vlan 11 4/1-48(VLAN 11 corresponds to subnet “11” ie 10.11.0.0)
Configuration of the VLAN Building Block
The VLAN building block appears on the right side (east block) of Figures 3a, 3b, and 3c. If dual-attached servers are used, then a VLAN trunking configuration is required within the server distribution block. The two distribution-layer switches are the root bridges of the even and odd numbered VLANs. UplinkFast is configured on the access-layer switches. BackboneFast is configured on the access and distribution switches for faster spanning tree recovery. For maximum determinism, VTP transparent mode is used and all VLANs are configured explicitly. HSRP at Layer 3 is configured to match the Layer 2 spanning tree configuration. This way, the HSRP primary gateway router for even-numbered subnets is also the spanning tree root for even-numbered VLANs. The HSRP primary gateway router for odd-numbered subnets is also the spanning tree root for odd-numbered VLANs.
Configuration of Catalyst 6000 Access Switch:
Set prompt ae6 (access layer, east block, Catalyst 6000) Set vtp domain east Set vtp mode transparent (use transparent mode, configure all VLANs explicitly) Set vlan 99 (used for out of band management) Set vlan 99 4/48 (last physical port on switch used for oob management) Set int sc0 99 172.29.196.51 255.255.254.0 (logical console port for oob management) Set ip route default 172.29.196.1 (gateway router in oob management network) set spantree uplinkfast enable (set uplinkfast on the access switch only) set spantree backbonefast enable (enable backbonefast on all switches in block) Set port channel 1/1-2 mode off (turn off channel negotiation on uplinks) (not using EtherChannel on this switch) set trunk 1/1-2 on 50,51,52,53 dot1q (dot1q VLAN trunking on uplinks, state VLANs explicitly) (these four VLANs are used in this building block) set vlan 50 2/1-48(VLAN 50 corresponds to subnet “50” ie 10.50.0.0) set vlan 51 3/1-48(VLAN 51 corresponds to subnet “51” ie 10.51.0.0)
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 3 of 26
Configuration of Catalyst 6500 Distribution Switch:
The ARP cache timeout on the MSFC is four hours. However, the Layer 2 CAM table times out in 300 seconds by default. This may result in some IP unicast traffic being flooded. In the following configuration the CAM timeout “agingtime” is set to 4hours*60min/hour*60sec/ min. = 14400 seconds to match the ARP cache timeout.
Set prompt det65 (distribution layer, east block, top, Catalyst 6500) Set vtp domain east Set vtp mode transparent (use transparent mode, configure all VLANs explicitly) Set vlan 99 (used for out of band management) Set vlan 99 3/48 (last physical port on switch used for oob management) Set int sc0 99 172.29.196.51 255.255.254.0 (logical console port for oob management) Set ip route default 172.29.196.1 (gateway router in oob management network) set cam agingtime 1-1000 14400 (set CAM timeout to 4 hours to match ARP timeout) set spantree backbonefast enable (enable backbonefast on all switches in block) Set port channel 1/1-2 mode off (turn off channel negotiation on non-Etherchannel links) set port channel 3/7-8 mode off set port channel 4/7-8 mode off set port channel 3/1-2 mode on (EtherChannel used for routed links to the core) set port channel 4/1-2 mode on (EtherChannel used for routed links to the core) set vlan 40 3/1,4/1 (routed etherchannel link VLAN40=10.40.0.0 spans two cards) set vlan 42 3/2,4/2 (routed etherchannel link VLAN42=10.42.0.0 spans two cards) set trunk 3/7 on 50,51,52,53 dot1q (set VLANs and dot1q trunking explicitly) (3/7-8 and 4/7-8 are uplinks to wiring closet switches) set trunk 3/8 on 50,51,52,53 dot1q set trunk 4/7 on 50,51,52,53 dot1q set trunk 4/8 on 50,51,52,53 dot1q set trunk 1/1 on 50,51,52,53 dot1q (this is the backup trunk to other distribution switch) set spantree root 50,52 (make this root bridge for even VLANs) set spantree root secondary 51,53 (make this backup root bridge odd VLANs) set trunk 3/1-2 off (no VLAN trunking on routed links to core) set trunk 4/1-2 off (no VLAN trunking on routed links to core)
As shown in Figures 3b and 3c, VLANs 50, 51, 52, and 53 correspond to the wiring closet subnets. VLANs 40 and 42 correspond to routed links to the core switches. Refer to section three of the test results for the corresponding MSFC (router) configuration.
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 4 of 26
Notes on Test Procedures
Test results are measured by recovery of multiple two-way PING traffic flows. In all cases, test traffic flows are between clients attached to the network, not to and from the switches within the network. We measure the outage when the failure is created in terms of seconds of lost traffic. Then we measure the outage when the failure is restored in terms of seconds of lost traffic. Results are rounded up to the next second, which is the limit of accuracy with one-second-hello protocols. If recovery is given as three seconds in the table, that means we measured two seconds and three seconds over several tests. The “small” routing table just reflects the subnets within the test bed. To stress routing protocol convergence, the “large” routing table includes 3000 static host routes injected into the test bed from the six Layer 3 switches. Summary of IP Routing with Small Routing Table
ctmsfc#sho ip route summary Route Source Networks connected 1 static 1 eigrp 1 0 internal 2 Total 4
Subnets 5 0 11 16
Overhead 336 56 616 1008
Memory (bytes) 864 144 1584 2328 4920
Summary of IP Routing with Large Routing Table
ctmsfc#sho ip ro sum Route Source Networks connected 1 static 1 eigrp 1 0 internal 2 Total 4
Subnets 5 500 2511 3016
Overhead 336 28056 140616 169008
Memory (bytes) 864 72144 361584 2328 436920
To stress the control plane (routing software running on the CPU) static routes are added to each of the routers (Layer 3 switches) in the test bed by Trivial File Transfer Protocol (TFTP). TFTP configuration is accomplished with the “configure network” command. The files on the TFTP server consist of a series of commands of the form:
ip route ip route ip route ip route … ip route end 10.0.1.001 10.0.1.002 10.0.1.003 10.0.1.004 255.255.255.255 255.255.255.255 255.255.255.255 255.255.255.255 null null null null 0 0 0 0 1 1 1 1
10.0.1.254 255.255.255.255 null 0 1
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 5 of 26
1. Layer 3 Distribution with Dual-Path Layer 2 Core—Catalyst 6500 Please refer to Figures 1a, 1b, and 1c for this section. Clients test1 through test8 were attached to the wiring closet switches in order to test network recovery times as different failures were induced. This campus design features redundant Layer 3 switches in each distribution layer building block and redundant Layer 2 switches in the core. The dual-path Layer 2 core consists of two separate switched VLANs with no loops and no VLAN trunks. The links into the core VLANs are native routed interfaces on the distribution-layer switches, and VLAN trunks are not used.
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 6 of 26
Configuration of MSFC: Client-Side Interface on HSRP Primary Distribution Switch
interface Vlan10 ip address 10.10.0.81 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 (set eigrp hello timer 1 second) ip hold-time eigrp 1 3 (set eigrp hold timer 3 seconds) standby 10 timers 1 3 (set HSRP hello timer 1 hold timer 3 seconds) (convention - HSRP group number 10 matches VLAN number) standby 10 priority 200 preempt delay 60 (this is the primary gateway router for subnet 10) (preempt delay 60 seconds allows EIGRP to stabilize before HSRP switches back upon power recovery) standby 10 ip 10.10.0.200 (10.10.0.200 is the HSRP gateway router address) standby 10 track Vlan31 75 standby 10 track Vlan32 75 (if you lose both links to the backbone, drop priority by 150 to initiate HSRP recovery)
Configuration of MSFC: Client-Side Interface on HSRP Secondary Distribution Switch
interface Vlan10 ip address 10.10.0.82 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 standby 10 timers 1 3 standby 10 priority 100 preempt delay 60 (HSRP secondary or backup gateway router for subnet 10) standby 10 ip 10.10.0.200 standby 10 track Vlan31 25 standby 10 track Vlan32 25
Additional MSFC Configuration—Interface to Backbone VLAN 31
interface Vlan31 ip address 10.31.0.81 255.255.0.0 no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3
Additional MSFC Configuration—Interface on Management VLAN
interface Vlan99 ip address 172.26.196.81 255.255.254.0 no ip directed-broadcast
Additional MSFC Configuration—EIGRP with Passive Interfaces to Wiring Closets
router eigrp 1 passive-interface passive-interface passive-interface passive-interface passive-interface network 10.0.0.0
Vlan10 Vlan11 Vlan12 Vlan13 Vlan99
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 7 of 26
2. Layer 3 Distribution with Layer 3 Core—Catalyst 6500 Please refer to Figures 2a, 2b, and 2c for this section. Clients test1 through test8 were attached to the wiring closet switches in order to test network recovery times as different failures were induced. Each distribution-layer switch has a single connection into the core, but each building block as a whole has redundant connectivity into the core. This campus design features redundant Layer 3 switches in the distribution layer. For building block “west” the redundant switches are dwt65 (distribution west top 6500) and dwb65 (distribution west bottom 6500). Each distribution switch has a two-port Gigabit EtherChannel connection to a Layer 3 switch in the core. The HSRP track is configured on dwt65 and dwb65 so that fast HSRP recovery will take place if the routed link to the core is broken. There is one important caveat with this design: because each distribution-layer switch has a single path into the core, a redundant routed path must be provided in case this fails. If the physical link breaks, then HSRP will handle the recovery. However, in the event of some logical failure of the MSFC within the core switch, HSRP will not be triggered. An example of this is when the MSFC in the core router is reloaded. Eliminate the passive interface command on two of the wiring closet VLAN interfaces to provide two routed backup paths between the distribution-layer switches.
Table 3 Results with Small Routing Table
Test
Wiring Closet Uplink
Failure
Time of Interruption
Recovery Mechanism
Fail Restore Distribution Switch Fail Restore Core Switch Fail Restore
4s 1s
HSRP HSRP
4s 1s
HSRP HSRP
4s 14s
HSRP (See note below)
Note: When power is restored to the core switch, the links out to the distribution layer come up before EIGRP has stabilized. Therefore, HSRP on the distribution switch preempts the primary gateway router function a few seconds before EIGRP on the distribution switch has a routing table built; hence the 14-second interruption as noted. The preempt delay command-line option refers to delay upon powerup of the HSRP switch itself, and does not affect this value.
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 8 of 26
Table 4 Results with Large Routing Table
Test
Wiring Closet Uplink
Failure
Time of Interruption
Recovery Mechanism
Fail Restore Distribution Switch Fail Restore Core Switch Fail Restore
4s 1s
HSRP HSRP
4s 1s
HSRP HSRP
4s 30s
HSRP (See note below)
Note: When power is restored to the core switch, the links out to the distribution layer come up before EIGRP has stabilized. Therefore, HSRP on the distribution switch preempts the primary gateway router function a few seconds before EIGRP on the distribution switch has a routing table built; hence the 30-second interruption as noted. The preempt delay command-line option refers to delay upon powerup of the switch running HSRP in the distribution layer, and is not effective when the core switch is powered up.
Configuration of MSFC: Client-Side Interface on HSRP Primary Distribution Switch
interface Vlan10 ip address 10.10.0.81 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 (set eigrp hello timer 1 second) ip hold-time eigrp 1 3 (set eigrp hold timer 3 seconds) standby 10 timers 1 3 (set HSRP hello timer 1 hold timer 3 seconds) (convention - HSRP group number 10 matches VLAN number) standby 10 priority 200 preempt delay 60 (this is the primary gateway router for subnet 10) (preempt delay 60 seconds allows EIGRP to stabilize before HSRP switches back upon power recovery) standby 10 ip 10.10.0.200 (10.10.0.200 is the HSRP gateway router address) standby 10 track Vlan20 150 (if you lose the link to the backbone, drop priority by 150 to initiate HSRP recovery)
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 9 of 26
Configuration of MSFC: Client-Side Interface on HSRP Secondary Distribution Switch
interface Vlan10 ip address 10.10.0.82 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 standby 10 timers 1 3 standby 10 priority 100 preempt delay 60 (HSRP secondary or backup gateway router for subnet 10) standby 10 ip 10.10.0.200 standby 10 track Vlan23 50
Additional MSFC Configuration—Any Routed Interface to Backbone
interface Vlan21 ip address 10.21.0.81 255.255.0.0 no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3
Additional MSFC Configuration—Interface on Management VLAN
interface Vlan99 ip address 172.26.196.81 255.255.254.0 no ip directed-broadcast
Additional MSFC Configuration—EIGRP with Passive Interfaces to Wiring Closets
router eigrp 1 passive-interface Vlan10 passive-interface Vlan11 (no passive interface on VLAN 12 or VLAN 13) (two VLANs are kept as redundant routed paths) passive-interface Vlan99 network 10.0.0.0
3. Layer 3 Distribution with Dual-Path Layer 3 Core—Catalyst 6500 Please refer to Figures 3a, 3b, and 3c for this section. Clients test1 through test8 were attached to the wiring closet switches in order to test network recovery times as different failures were induced. This design has the highest level of redundancy and the highest link capacity into the core. This campus design features redundant Layer 3 switches in the distribution layer. For building block “west” the redundant switches are dwt65 (distribution west top 6500) and dwb65 (distribution west bottom 6500). Each distribution switch has redundant two-port Gigabit EtherChannel connections to both Layer 3 switches in the core. The HSRP track is configured on dwt65 and dwb65 so that fast HSRP recovery will take place if both routed links to the core are broken.
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 10 of 26
Table 5 Testing with the Standard Building Block—Small Routing Table
Test
Wiring Closet Uplink
Failure
Time of Interruption
Recovery Mechanism
Fail Restore Distribution Switch Fail Restore Core Switch Fail Restore
3s 0s
HSRP HSRP
3s 0s
HSRP HSRP
0s 6s
Dual-path redundancy EIGRP
Table 6 Testing with the Standard Building Block—Large Routing Table
Test
Wiring Closet Uplink
Failure
Time of Interruption
Recovery Mechanism
Fail Restore Distribution Switch Fail Restore Core Switch Fail Restore
3s 0s
HSRP HSRP
3s 0s
HSRP HSRP
0s 20s
Dual-path redundancy EIGRP
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 11 of 26
Table 7 Testing with the Standard Building Block—Small Routing Table
Test
Wiring Closet Uplink
Failure
Time of Interruption
Recovery Mechanism
Fail Restore Distribution Switch Fail
2s 0s
UplinkFast UplinkFast
3s
UplinkFast/HSRP
Restore Core Switch Fail Restore VLAN Backup Trunk Fail Restore
15s
(See note below)
3s 0s
Dual-path redundancy Dual-path redundancy
31s 28s
Spanning tree Spanning tree
Note: Restoration of the distribution-layer switch causes a 15-second outage. This is a result of the fact that the Layer 2 and Layer 3 path determination functions on the switch recover at slightly different moments. When the routing table is increased to 3000 routes in the next test, this anomaly is eliminated.
Table 8 Testing with the VLAN Building Block—Large Routing Table
Test
Wiring Closet Uplink
Failure
Time of Interruption
Recovery Mechanism
Fail Restore Distribution Switch Fail
2s 0s
UplinkFast UplinkFast
3s
UplinkFast/HSRP
Restore Core Switch Fail Restore VLAN Backup Trunk Fail Restore
5s
(See note below)
3s 0s
Dual-path redundancy Dual-path redundancy
31s 28s
Spanning tree Spanning tree
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 12 of 26
Note: When the distribution switch is restored, a five-second outage results because the Layer 2 topology stabilizes at a slightly different moment than the Layer 3 topology. With the large routing table this difference is reduced from the previous test.
Configuration of MSFC: Client-Side Interface on Distribution Switch—HSRP Primary
interface Vlan10 ip address 10.10.0.81 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 (set eigrp hello timer 1 second) ip hold-time eigrp 1 3 (set eigrp hold timer 3 seconds) standby 10 timers 1 3 (set HSRP hello timer 1 hold timer 3 seconds) (convention - HSRP group number 10 matches VLAN number) standby 10 priority 200 preempt delay 60 (this is the primary gateway router for subnet 10) (preempt delay 60 seconds allows EIGRP to stabilize before HSRP switches back upon power recovery) standby 10 ip 10.10.0.200 (10.10.0.200 is the HSRP gateway router address) standby 10 track Vlan20 75 standby 10 track Vlan21 75 (if you lose both links to the backbone, drop priority by 150 to initiate HSRP recovery)
Configuration of MSFC: Client-Side Interface on Distribution Switch—HSRP Secondary
interface Vlan10 ip address 10.10.0.82 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 standby 10 timers 1 3 standby 10 priority 100 preempt delay 60 (HSRP secondary or backup gateway router for subnet 10) standby 10 ip 10.10.0.200 standby 10 track Vlan22 25 standby 10 track Vlan23 25 (track both links to the backbone)
Additional MSFC Configuration—Any Routed Interface to Backbone
interface Vlan21 ip address 10.21.0.81 255.255.0.0 no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3
Additional MSFC Configuration—Interface on Management VLAN
interface Vlan99 ip address 172.26.196.81 255.255.254.0 no ip directed-broadcast
Additional MSFC Configuration—EIGRP with Passive Interfaces to Wiring Closets
router eigrp 1 passive-interface passive-interface passive-interface passive-interface passive-interface network 10.0.0.0
Vlan10 Vlan11 Vlan12 Vlan13 Vlan99
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 13 of 26
4. Layer 3 Distribution with Layer 3 Core—Catalyst 8540 Please refer to Figures A1a, A1b, and A1c for this section. Clients target1 through target6 were attached to the wiring closet switches in order to test network recovery times as different failures were induced. Each distribution-layer switch has a single connection into the core, but each building block as a whole has redundant connectivity into the core. This campus design features redundant Layer 3 switches in the distribution layer. For building block “left” the redundant switches are dlt85 (distribution left top 8540) and dlb85 (distribution left bottom 8540). Each distribution switch has a two-port Gigabit EtherChannel connection to a Layer 3 switch in the core. The HSRP track is configured on dlt85 and dlb85 so that fast HSRP recovery will take place if the routed link to the core is broken.
Table 9 Results with Small Routing Table
Test
Wiring Closet Uplink
Failure
Time of Interruption
Recovery Mechanism
Fail Restore Distribution Switch Fail Restore Core Switch Fail Restore
4s 1s
HSRP HSRP
3s 1s
HSRP HSRP
6s 3s
EIGRP HSRP
Table 10 Results with Large Routing Table
Test
Wiring Closet Uplink
Failure
Time of Interruption
Recovery Mechanism
Fail Restore Distribution Switch Fail Restore Core Switch Fail Restore
4s 1s
HSRP HSRP
3s 2s
HSRP HSRP
7s 3s
EIGRP HSRP
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 14 of 26
Configuration Details of HSRP Primary Distribution Router dlt85
interface Port-channel1 ip address 10.70.0.53 255.255.0.0 no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 hold-queue 300 in ! interface Port-channel2 ip address 10.71.0.53 255.255.0.0 no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 hold-queue 300 in ! interface GigabitEthernet0/0/1 no ip address no ip directed-broadcast channel-group 1 ! interface GigabitEthernet1/0/0 ip address 10.60.0.53 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 standby timers 1 3 standby priority 200 standby preempt standby ip 10.60.0.200 standby track Port-channel1 150 ! interface GigabitEthernet1/0/1 no ip address no ip directed-broadcast channel-group 1 ! interface GigabitEthernet2/0/0 ip address 10.61.0.53 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 standby timers 1 3 standby priority 200 standby preempt standby ip 10.61.0.200 standby track Port-channel1 150 ! interface GigabitEthernet2/0/1 no ip address no ip directed-broadcast channel-group 2 ! interface GigabitEthernet3/0/0 ip address 10.62.0.53 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 15 of 26
standby standby standby standby standby
timers 1 3 priority 200 preempt ip 10.62.0.200 track Port-channel1 150
! interface GigabitEthernet3/0/1 no ip address no ip directed-broadcast channel-group 2 ! interface Ethernet0 (management interface) ip address 172.26.196.53 255.255.254.0 no ip directed-broadcast ! ……… ! router eigrp 1 passive-interface GigabitEthernet0/0/0 passive-interface GigabitEthernet1/0/0 passive-interface GigabitEthernet2/0/0 passive-interface GigabitEthernet3/0/0 network 10.0.0.0 ! ……… end
5. Layer 3 Distribution with Dual-Path Layer 3 Core—Catalyst 8540 Please refer to Figures A2a, A2b, and A2c for this section. Clients target1 through target6 were attached to the wiring closet switches in order to test network recovery times as different failures were induced. The dual-path design has the highest level of redundancy and the highest link capacity into the core. This campus design features redundant Layer 3 switches in the distribution layer. For building block “left” the redundant switches are dlt85 (distribution left top 8540) and dlb85 (distribution left bottom 8540). Each distribution switch has redundant two-port Gigabit EtherChannel connections to both Layer 3 switches in the core. The HSRP track is configured on dlt85 and dlb85 so that fast HSRP recovery will take place if both routed links to the core are broken.
Table 11 Results with Small Routing Table
Test
Wiring Closet Uplink
Failure
Time of Interruption
Recovery Mechanism
Fail Restore Distribution Switch Fail Restore Core Switch Fail Restore
3s 0s
HSRP HSRP
3s 0s
HSRP HSRP
3s 1s
EIGRP EIGRP
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 16 of 26
Table 12 Results with Large Routing Table
Test
Wiring Closet Uplink
Failure
Time of Interruption
Recovery Mechanism
Fail Restore Distribution Switch Fail Restore Core Switch Fail Restore
3s 0s
HSRP HSRP
3s 11s
HSRP (See note below)
10s 1s
EIGRP EIGRP
Note: Restoring the distribution switch results in an outage between the time that HSRP switches and the EIGRP routing table is built. This can be corrected by adding a “preempt delay” as in the following command:
standby 10 priority 100 preempt delay 60
Configuration details of HSRP primary distribution router dlt85
interface Port-channel1 ip address 10.70.0.53 255.255.0.0 no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 hold-queue 300 in ! interface Port-channel2 ip address 10.71.0.53 255.255.0.0 no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 hold-queue 300 in ! interface GigabitEthernet0/0/1 no ip address no ip directed-broadcast channel-group 1 ! interface GigabitEthernet1/0/0 ip address 10.60.0.53 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 standby timers 1 3 standby priority 200 standby preempt standby ip 10.60.0.200 standby track Port-channel1 75 standby track Port-channel2 75 !
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 17 of 26
interface GigabitEthernet1/0/1 no ip address no ip directed-broadcast channel-group 1 ! interface GigabitEthernet2/0/0 ip address 10.61.0.53 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 standby timers 1 3 standby priority 200 standby preempt standby ip 10.61.0.200 standby track Port-channel1 75 standby track Port-channel2 75 ! interface GigabitEthernet2/0/1 no ip address no ip directed-broadcast channel-group 2 ! interface GigabitEthernet3/0/0 ip address 10.62.0.53 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 standby timers 1 3 standby priority 200 standby preempt standby ip 10.62.0.200 standby track Port-channel1 75 standby track Port-channel2 75 ! interface GigabitEthernet3/0/1 no ip address no ip directed-broadcast channel-group 2 ! interface Ethernet0 ip address 172.26.196.53 255.255.254.0 no ip directed-broadcast ! ……… ! router eigrp 1 passive-interface GigabitEthernet0/0/0 passive-interface GigabitEthernet1/0/0 passive-interface GigabitEthernet2/0/0 passive-interface GigabitEthernet3/0/0 network 10.0.0.0 ! ……… end
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 18 of 26
1A: Dual-Path Layer 2 Core—Layout, Catalyst 6500
aw3 West Block Standard Test1 aw4 dwt65 ct65 det65 East Block VLAN
ae3
ae4
Test4
Test2
aw5
ae5
Test5
Test3
aw6 dwb65 cb65 deb65
ae6
Test6
Access
Distribution
Core
Distribution
Access
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
1B: Dual-Path Layer 2 Core—Logical, Catalyst 6500
aw3 10.10.0.101 10.10.0.0 172.26.196.11 West Block Standard East Block VLAN
ae3 10.50.0.101 10.5x.0.0 172.26.196.21
aw4 Test1 10.11.0.101 10.11.0.0 172.26.196.12
dwt65
ct65
det65
10.x.0.25 10.x.0.29 10.x.0.200 Pri 10.31.0.0 10.x.0.200 Pri 172.26.196.25 172.26.196.27 172.26.196.29
Test4 10.51.0.101 10.5x.0.0 172.26.196.22
ae4
aw5 Test2 10.12.0.101 10.12.0.0 172.26.196.13 10.x.0.26 10.x.0.30 10.32.0.0 10.x.0.200 Sec 10.x.0.200 Sec 172.26.196.28 172.26.196.26 172.26.196.30 dwb65 cb65 deb65
ae5 10.5x.0.0 172.26.196.23
Test5 10.52.0.101
aw6 Test3 10.13.0.101 10.13.0.0 172.26.196.14 Access
Test6 10.53.0.101 10.5x.0.0 172.26.196.24 Access
ae6
Distribution
Core
Distribution
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 19 of 26
1C: Dual-Path Layer 2 Core—Backbone, Catalyst 6500
West Block dwt65 VLAN 31 ct65 10.31.0.0
East Block det65 VLAN 31
VLAN 32
VLAN 32
VLAN 31 10.32.0.0 VLAN 32 dwb65 cb65
VLAN 31
VLAN 32 deb65
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
2A: Layer 3 Core—Layout, Catalyst 6500
aw3 West Block Standard Test1 aw4 dwt65 ct65 det65 East Block VLAN
ae3
ae4
Test4
Test2
aw5
ae5
Test5
Test3
aw6 dwb65 cb65 deb65
ae6
Test6
Access
Distribution
Core
Distribution
Access
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 20 of 26
2B: Layer 3 Core—Logical, Catalyst 6500
aw3 10.10.0.101 10.10.0.0 172.26.196.11 West Block Standard East Block VLAN
ae3 10.50.0.101 10.5x.0.0 172.26.196.21
aw4 Test1 10.11.0.101 10.11.0.0 172.26.196.12
dwt65
ct65
det65
10.x.0.25 10.x.0.29 10.x.0.200 Pri 10.x.0.27 10.x.0.200 Pri 172.26.196.25 172.26.196.27 172.26.196.29
Test4 10.51.0.101 10.5x.0.0 172.26.196.22
ae4
aw5 Test2 10.12.0.101 10.12.0.0 172.26.196.13 10.x.0.26 10.x.0.30 10.x.0.28 10.x.0.200 Sec 10.x.0.200 Sec 172.26.196.28 172.26.196.26 172.26.196.30 dwb65 cb65 deb65
Test5 10.52.0.101 10.5x.0.0 172.26.196.23
ae5
aw6 Test3 10.13.0.101 10.13.0.0 172.26.196.14 Access
ae6 10.5x.0.0 172.26.196.24 Access
Test6 10.53.0.101
Distribution
Core
Distribution
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
2C: Layer 3 Core—Backbone, Catalyst 6500
West Block dwt65 10.20.0.0 VLAN 20 VLAN 20 VLAN 30 VLAN 40 ct65
East Block det65 10.40.0.0 VLAN 40
10.30.0.0
VLAN 30 10.23.0.0 VLAN 23 dwb65 VLAN 23 cb65 VLAN 43 10.43.0.0 VLAN 43 deb65
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 21 of 26
3A: Dual-Path Layer 3 Core—Layout, Catalyst 6500
aw3 West Block Standard Test1 aw4 dwt65 ct65 det65 East Block VLAN
ae3
ae4
Test4
Test2
aw5
ae5
Test5
Test3
aw6 dwb65 cb65 deb65
ae6
Test6
Access
Distribution
Core
Distribution
Access
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
3B: Dual-Path Layer 3 Core—Logical, Catalyst 6500
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 22 of 26
3C: Dual-Path Layer 3 Core—Backbone, Catalyst 6500
West Block dwt65 VLAN 20 VLAN 21 10.21.0.0 VLAN 30 10.20.0.0 VLAN 20 VLAN 22 ct65 VLAN 40 VLAN 41
East Block 10.40.0.0 VLAN 40
det65
VLAN 42 10.42.0.0
10.30.0.0 10.22.0.0 VLAN 30 VLAN 22 10.23.0.0 VLAN 23 dwb65 Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
A1A: Layer 3 Core—Layout, Catalyst 8540
10.41.0.0 VLAN 41
VLAN 21 VLAN 23 cb65
VLAN 42 VLAN 43
10.43.0.0 VLAN 43 deb65
Left Block
Right Block
al4 dlt85 ct85 drt85
ar4
Target1
al5
ar5
Target3
Target2
al6 dlb85 Access Distribution cb85 Core drb85
ar6
Target4
Distribution
Access
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 23 of 26
A1B: Layer 3 Core—Logical, Catalyst 8540
Left Block
Right Block
al4 10.60.0.101 10.60.0.0 172.26.196.50 dlt85 ct65 drt85 10.x.0.53 10.x.0.59 10.x.0.200 Pri 10.x.0.57 10.x.0.200 Pri 172.26.196.53 172.26.196.57 172.26.196.59
ae4 10.100.0.101 10.100.0.0 172.26.196.61
al5 Target1 10.61.0.101 10.61.0.0 172.26.196.51 10.x.0.54 10.x.0.60 10.x.0.58 10.x.0.200 Sec 10.x.0.200 Sec 172.26.196.58 172.26.196.54 172.26.196.60 dlb85 cb85 drb85
ae5 10.101.0.0 172.26.196.62
Target3 10.101.0.101
al6 Target2 10.62.0.101 10.62.0.0 172.26.196.52 Access
Target4 10.102.0.101 10.102.0.0 172.26.196.63 Access
ae6
Distribution
Core
Distribution
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
A1C: Layer 3 Core—Topology, Catalyst 8540
Left Block dlt85 10.70.0.0 Port-chan 1 int gig 0/0/1 int gig 1/0/1 Port-chan 1 int gig 0/0/0 int gig 1/0/0 Port-chan 5 int gig 9/0/0 int gig 9/0/1 10.80.0.0 Port-chan 3 int gig 0/0/1 int gig 1/0/1 ct85
Right Block drt85 10.90.0.0 Port-chan 1 int gig 0/0/0 int gig 1/0/0
Port-chan 5 int gig 9/0/0 int gig 9/0/1 10.73.0.0 Port-chan 2 int gig 2/0/1 int gig 3/0/1 Port-chan 2 int gig 2/0/0 int gig 3/0/0 Port-chan 4 int gig 2/0/1 int gig 3/0/1 10.93.0.0 Port-chan 2 int gig 2/0/0 int gig 3/0/0
dlb85
cb85
drb85
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 24 of 26
A2A: Dual-Path Layer 3 Core—Layout, Catalyst 8540
Left Block
Right Block
al4 dlt85 ct85 drt85
ar4
Target1
al5
ar5
Target3
Target2
al6 dlb85 cb85 drb85
ae6
Target4
Access
Distribution
Core
Distribution
Access
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
A2B: Dual-Path Layer 3 Core—Logical, Catalyst 8540
Left Block
Right Block
al4 10.60.0.101 10.60.0.0 172.26.196.50 dlt85 ct85 drt85 10.x.0.53 10.x.0.59 10.x.0.200 Pri 10.x.0.57 10.x.0.200 Pri 172.26.196.53 172.26.196.57 172.26.196.59
ar4 10.100.0.101 10.100.0.0 172.26.196.61
al5 Target1 10.61.0.101 10.61.0.0 172.26.196.51 10.x.0.54 10.x.0.60 10.x.0.58 10.x.0.200 Sec 10.x.0.200 Sec 172.26.196.58 172.26.196.54 172.26.196.60 dlb85 cb85 drb85
ar5 10.101.0.0 172.26.196.62
Target3 10.101.0.101
al6 Target2 10.62.0.101 10.62.0.0 172.26.196.52 Access
ar6 10.102.0.0 172.26.196.63 Access
Target4 10.102.0.101
Distribution
Core
Distribution
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 25 of 26
A2C: Dual-Path Layer 3 Core—Backbone, Catalyst 8540
Left Block dlt85 Port 1 Port 2 10.71.0.0 Port 5 10.70.0.0 Port 1 Port 2 ct85 Port 3 Port 4
Right Block 10.90.0.0 VLAN 40
drt85
Port 2 10.92.0.0
10.80.0.0 10.72.0.0 Port 5 Port 1 10.73.0.0 Port 2 dlb85 Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual Port 1 Port 2 cb85 Port 3 Port 4 10.93.0.0 Port 2 drb85
10.91.0.0 Port 1
Conclusion
Choose a deterministic, structured design model to achieve high availability in your enterprise network. Apply redundancy in the mission-critical parts of the network. Scale the enterprise network to the size required by choosing the appropriate building block model and combining with the right backbone model. For a better understanding of the different choices referred to in this paper refer to the paper Gigabit Campus Network Design—Principles and Architecture.
Geoff Haviland ([email protected])—Network Design Engineer.
Corporate Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 526-4100
European Headquarters Cisco Systems Europe s.a.r.l. Parc Evolic, Batiment L1/L2 16 Avenue du Quebec Villebon, BP 706 91961 Courtaboeuf Cedex France http://www-europe.cisco.com Tel: 33 1 69 18 61 00 Fax: 33 1 69 28 83 26
Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-7660 Fax: 408 527-0883
Asia Headquarters Nihon Cisco Systems K.K. Fuji Building, 9th Floor 3-2-3 Marunouchi Chiyoda-ku, Tokyo 100 Japan http://www.cisco.com Tel: 81 3 5219 6250 Fax: 81 3 5219 6001
Cisco Systems has more than 200 offices in the following countries. Addresses, phone numbers, and fax numbers are listed on the
C i s c o C o n n e c t i o n O n l i n e We b s i t e a t h t t p : / / w w w. c i s c o . c o m / o f f i c e s .
Argentina • Australia • Austria • Belgium • Brazil • Canada • Chile • China • Colombia • Costa Rica • Croatia • Czech Republic • Denmark • Dubai, UAE Finland • France • Germany • Greece • Hong Kong • Hungary • India • Indonesia • Ireland • Israel • Italy • Japan • Korea • Luxembourg • Malaysia Mexico • The Netherlands • New Zealand • Norway • Peru • Philippines • Poland • Portugal • Puerto Rico • Romania • Russia • Saudi Arabia • Singapore Slovakia • Slovenia • South Africa • Spain • Sweden • Switzerland • Taiwan • Thailand • Turkey • Ukraine • United Kingdom • United States • Venezuela
Copyright © 2000 Cisco Systems, Inc. All rights reserved. Printed in the USA. Catalyst, Cisco, Cisco IOS, Cisco Systems, and the Cisco Systems logo are registered trademarks of Cisco Systems, Inc. or its affiliates in the U.S. and certain other countries. All other trademarks mentioned in this document are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any of its resellers. (9912R) 02/00 BW5858
Gigabit Campus Design
Configuration and Recovery Analysis
Introduction
Some of the most useful variations of the large-scale multilayer campus internetwork design were tested for failure recovery. Please refer to the companion document Gigabit Campus Design—Principles and Architecture for an explanation of several implementations of the multilayer design. In all cases, Gigabit EtherChannel® links were used between distribution-layer switches and core-layer switches. Redundant Gigabit Ethernet uplinks were used to connect the access-layer switches to the distribution-layer switches. The Catalyst® 4000, 5500, and 6000 family switches were used in the wiring closet configurations. The Catalyst 6500 with Multilayer Switch Feature Card (MSFC) and Catalyst 8540 were both used as the Layer 3 switch. The Catalyst 6500 is used as the Layer 2 switch in the core. This paper documents configuration and recovery with three fundamental designs: 1. Layer 3 distribution with dual-path Layer 2 core (good) a) Catalyst 6500 standard building block 2. Layer 3 distribution with Layer 3 core (better) a) Catalyst 6500 standard building block 3. Layer 3 distribution with dual-path Layer 3 core (best) a) Catalyst 6500 standard building block b) Catalyst 6500 virtual LAN(VLAN) building block The following two design variations are described in the appendix: A1. Layer 3 distribution with Layer 3 core a) Catalyst 8540 standard building block A2. Layer 3 distribution with dual-path Layer 3 core a) Catalyst 8540 standard building block Layer 2 spanning-tree loops were eliminated in the core and in the standard building block. Avoidance of spanning-tree loops in the core is important for high availability. Within the core and the standard building block all Layer 3 switches were configured with native routed interfaces. VLAN trunks were only used within the VLAN building block. The VLAN building block is typically used in a server farm to provide for redundant server connection with dual Network Interface Cards (NICs). The Enhanced Interior Gateway Routing Protocol (EIGRP) was tuned for fast convergence with a one-second-hello timer and a three-second-hold timer. The Hot Standby Routing Protocol (HSRP) was tuned for fast convergence with a one-second-hello timer and a three-second-hold timer. When using protocol hello timers of one second, the limit of accuracy for recovery measurement is about one second.
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 1 of 26
Thus, all results are rounded to units of whole seconds, representing the worst case. Network recovery times were very predictable, with some variation due to the state of things such as the Address Resolution Protocol (ARP) cache. OSPF was configured with the same timer values as EIGRP, with similar results. Every router and switch in the network was provided an out-of-band Ethernet management interface on VLAN 99. One Ethernet port on each device was configured in VLAN 99 and wired outside the test network to a separate switched management network. In addition to the management IP address on VLAN 99, we also configured Cisco 2511 terminal servers with RJ-45 serial cables to access the console port of every router and switch in the layout. With this arrangement, we have two out-of-band paths to every switch and router, as well as a number of in-band paths. Wherever possible Gigabit EtherChannel was configured with ports on two different cards to increase availability. For example, port-channel 1 on CT85 (core top 8540) used interfaces gigabit 0/0/0 and gigabit 1/0/0, which are the first physical ports on two different cards. EtherChannel recovery was not tested, as it is much faster than one second and falls below the measurement threshold of this testing.
Configuration of the Standard Building Block
The standard building block appears on the left side (west block) of Figures 3a, 3b, and 3c. Configuration of the standard building block is very simple because all spanning-tree loops are eliminated. Hence, there is no Layer 2 tuning, such as selecting the best root switch for a given VLAN. No VLAN trunks are used and uplinks are connected to native routed interfaces on the Layer 3 switches in the distribution layer.
Catalyst 6000 Access Switch: Standard Building Block
Set prompt aw6 (access layer, west block, Catalyst 6000) Set vtp domain west Set vtp mode transparent (no VLAN trunks, use transparent mode) Set vlan 99 (used for out of band management) Set vlan 99 3/48 (last physical port on switch used for oob management) Set int sc0 99 172.29.196.51 255.255.254.0 (logical console port for oob management) Set ip route default 172.29.196.1 (gateway router in oob management network) Set port channel 1/1-2 mode off (turn off channel negotiation on uplinks) (not using EtherChannel in this configuration) set trunk 1/1-2 off(turn off VLAN trunking on uplinks) set vlan 10(VLAN 10 corresponds to subnet “10” ie 10.10.0.0) set vlan 10 1/1-2(all other ports are part of VLAN 10) set vlan 10 3/1-47
We disable channel negotiation on the uplinks to make connection faster after a failure has been restored. This is appropriate because the uplinks are routed connections to Layer 3 switches in the distribution layer, not Layer 2 connections that require spanning tree. (Nothing in the configuration indicates that all clients and uplinks attached to the switch are in subnet “10” (10.10.0.0) with mask 255.255.0.0. VLAN 10 is configured everywhere to match subnet “10” [10.10.0.0]).
Configuration of the Standard Building Block with Load Balancing
To achieve load balancing with the standard building block, a couple of changes are required. Configure two VLANs (two subnets) on a wiring closet switch and use VLAN trunks for uplinks. For example, on switch aw6, configure VLAN 10 and VLAN 11. All outbound VLAN 10 traffic will take one uplink to the HSRP primary gateway router for subnet 10.10.0.0. All outbound VLAN 11 traffic will take the other uplink to the HSRP primary gateway router for subnet 10.11.0.0. This load balancing configuration maintains all the advantages of the standard building block and is only marginally more complex.
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 2 of 26
Catalyst 6000 Access Switch: Standard Building Block with Load Balancing
Set prompt aw6 (access layer, west block, Catalyst 6000) Set vtp domain west Set vtp mode transparent (use transparent mode, configure VLANs explicitly) Set vlan 99 (used for out of band management) Set vlan 99 5/48 (last physical port on switch used for oob management) Set int sc0 99 172.29.196.51 255.255.254.0 (logical console port for oob management) Set ip route default 172.29.196.1 (gateway router in oob management network) Set port channel 1/1-2 mode off (turn off channel negotiation on uplinks) (not using EtherChannel on this switch) set trunk 1/1-2 on 10,11 dot1q (VLAN trunking on uplinks, for VLAN 10 and VLAN 11) set vlan 10 3/1-48(VLAN 10 corresponds to subnet “10” ie 10.10.0.0) set vlan 11 4/1-48(VLAN 11 corresponds to subnet “11” ie 10.11.0.0)
Configuration of the VLAN Building Block
The VLAN building block appears on the right side (east block) of Figures 3a, 3b, and 3c. If dual-attached servers are used, then a VLAN trunking configuration is required within the server distribution block. The two distribution-layer switches are the root bridges of the even and odd numbered VLANs. UplinkFast is configured on the access-layer switches. BackboneFast is configured on the access and distribution switches for faster spanning tree recovery. For maximum determinism, VTP transparent mode is used and all VLANs are configured explicitly. HSRP at Layer 3 is configured to match the Layer 2 spanning tree configuration. This way, the HSRP primary gateway router for even-numbered subnets is also the spanning tree root for even-numbered VLANs. The HSRP primary gateway router for odd-numbered subnets is also the spanning tree root for odd-numbered VLANs.
Configuration of Catalyst 6000 Access Switch:
Set prompt ae6 (access layer, east block, Catalyst 6000) Set vtp domain east Set vtp mode transparent (use transparent mode, configure all VLANs explicitly) Set vlan 99 (used for out of band management) Set vlan 99 4/48 (last physical port on switch used for oob management) Set int sc0 99 172.29.196.51 255.255.254.0 (logical console port for oob management) Set ip route default 172.29.196.1 (gateway router in oob management network) set spantree uplinkfast enable (set uplinkfast on the access switch only) set spantree backbonefast enable (enable backbonefast on all switches in block) Set port channel 1/1-2 mode off (turn off channel negotiation on uplinks) (not using EtherChannel on this switch) set trunk 1/1-2 on 50,51,52,53 dot1q (dot1q VLAN trunking on uplinks, state VLANs explicitly) (these four VLANs are used in this building block) set vlan 50 2/1-48(VLAN 50 corresponds to subnet “50” ie 10.50.0.0) set vlan 51 3/1-48(VLAN 51 corresponds to subnet “51” ie 10.51.0.0)
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 3 of 26
Configuration of Catalyst 6500 Distribution Switch:
The ARP cache timeout on the MSFC is four hours. However, the Layer 2 CAM table times out in 300 seconds by default. This may result in some IP unicast traffic being flooded. In the following configuration the CAM timeout “agingtime” is set to 4hours*60min/hour*60sec/ min. = 14400 seconds to match the ARP cache timeout.
Set prompt det65 (distribution layer, east block, top, Catalyst 6500) Set vtp domain east Set vtp mode transparent (use transparent mode, configure all VLANs explicitly) Set vlan 99 (used for out of band management) Set vlan 99 3/48 (last physical port on switch used for oob management) Set int sc0 99 172.29.196.51 255.255.254.0 (logical console port for oob management) Set ip route default 172.29.196.1 (gateway router in oob management network) set cam agingtime 1-1000 14400 (set CAM timeout to 4 hours to match ARP timeout) set spantree backbonefast enable (enable backbonefast on all switches in block) Set port channel 1/1-2 mode off (turn off channel negotiation on non-Etherchannel links) set port channel 3/7-8 mode off set port channel 4/7-8 mode off set port channel 3/1-2 mode on (EtherChannel used for routed links to the core) set port channel 4/1-2 mode on (EtherChannel used for routed links to the core) set vlan 40 3/1,4/1 (routed etherchannel link VLAN40=10.40.0.0 spans two cards) set vlan 42 3/2,4/2 (routed etherchannel link VLAN42=10.42.0.0 spans two cards) set trunk 3/7 on 50,51,52,53 dot1q (set VLANs and dot1q trunking explicitly) (3/7-8 and 4/7-8 are uplinks to wiring closet switches) set trunk 3/8 on 50,51,52,53 dot1q set trunk 4/7 on 50,51,52,53 dot1q set trunk 4/8 on 50,51,52,53 dot1q set trunk 1/1 on 50,51,52,53 dot1q (this is the backup trunk to other distribution switch) set spantree root 50,52 (make this root bridge for even VLANs) set spantree root secondary 51,53 (make this backup root bridge odd VLANs) set trunk 3/1-2 off (no VLAN trunking on routed links to core) set trunk 4/1-2 off (no VLAN trunking on routed links to core)
As shown in Figures 3b and 3c, VLANs 50, 51, 52, and 53 correspond to the wiring closet subnets. VLANs 40 and 42 correspond to routed links to the core switches. Refer to section three of the test results for the corresponding MSFC (router) configuration.
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 4 of 26
Notes on Test Procedures
Test results are measured by recovery of multiple two-way PING traffic flows. In all cases, test traffic flows are between clients attached to the network, not to and from the switches within the network. We measure the outage when the failure is created in terms of seconds of lost traffic. Then we measure the outage when the failure is restored in terms of seconds of lost traffic. Results are rounded up to the next second, which is the limit of accuracy with one-second-hello protocols. If recovery is given as three seconds in the table, that means we measured two seconds and three seconds over several tests. The “small” routing table just reflects the subnets within the test bed. To stress routing protocol convergence, the “large” routing table includes 3000 static host routes injected into the test bed from the six Layer 3 switches. Summary of IP Routing with Small Routing Table
ctmsfc#sho ip route summary Route Source Networks connected 1 static 1 eigrp 1 0 internal 2 Total 4
Subnets 5 0 11 16
Overhead 336 56 616 1008
Memory (bytes) 864 144 1584 2328 4920
Summary of IP Routing with Large Routing Table
ctmsfc#sho ip ro sum Route Source Networks connected 1 static 1 eigrp 1 0 internal 2 Total 4
Subnets 5 500 2511 3016
Overhead 336 28056 140616 169008
Memory (bytes) 864 72144 361584 2328 436920
To stress the control plane (routing software running on the CPU) static routes are added to each of the routers (Layer 3 switches) in the test bed by Trivial File Transfer Protocol (TFTP). TFTP configuration is accomplished with the “configure network” command. The files on the TFTP server consist of a series of commands of the form:
ip route ip route ip route ip route … ip route end 10.0.1.001 10.0.1.002 10.0.1.003 10.0.1.004 255.255.255.255 255.255.255.255 255.255.255.255 255.255.255.255 null null null null 0 0 0 0 1 1 1 1
10.0.1.254 255.255.255.255 null 0 1
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 5 of 26
1. Layer 3 Distribution with Dual-Path Layer 2 Core—Catalyst 6500 Please refer to Figures 1a, 1b, and 1c for this section. Clients test1 through test8 were attached to the wiring closet switches in order to test network recovery times as different failures were induced. This campus design features redundant Layer 3 switches in each distribution layer building block and redundant Layer 2 switches in the core. The dual-path Layer 2 core consists of two separate switched VLANs with no loops and no VLAN trunks. The links into the core VLANs are native routed interfaces on the distribution-layer switches, and VLAN trunks are not used.
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 6 of 26
Configuration of MSFC: Client-Side Interface on HSRP Primary Distribution Switch
interface Vlan10 ip address 10.10.0.81 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 (set eigrp hello timer 1 second) ip hold-time eigrp 1 3 (set eigrp hold timer 3 seconds) standby 10 timers 1 3 (set HSRP hello timer 1 hold timer 3 seconds) (convention - HSRP group number 10 matches VLAN number) standby 10 priority 200 preempt delay 60 (this is the primary gateway router for subnet 10) (preempt delay 60 seconds allows EIGRP to stabilize before HSRP switches back upon power recovery) standby 10 ip 10.10.0.200 (10.10.0.200 is the HSRP gateway router address) standby 10 track Vlan31 75 standby 10 track Vlan32 75 (if you lose both links to the backbone, drop priority by 150 to initiate HSRP recovery)
Configuration of MSFC: Client-Side Interface on HSRP Secondary Distribution Switch
interface Vlan10 ip address 10.10.0.82 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 standby 10 timers 1 3 standby 10 priority 100 preempt delay 60 (HSRP secondary or backup gateway router for subnet 10) standby 10 ip 10.10.0.200 standby 10 track Vlan31 25 standby 10 track Vlan32 25
Additional MSFC Configuration—Interface to Backbone VLAN 31
interface Vlan31 ip address 10.31.0.81 255.255.0.0 no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3
Additional MSFC Configuration—Interface on Management VLAN
interface Vlan99 ip address 172.26.196.81 255.255.254.0 no ip directed-broadcast
Additional MSFC Configuration—EIGRP with Passive Interfaces to Wiring Closets
router eigrp 1 passive-interface passive-interface passive-interface passive-interface passive-interface network 10.0.0.0
Vlan10 Vlan11 Vlan12 Vlan13 Vlan99
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 7 of 26
2. Layer 3 Distribution with Layer 3 Core—Catalyst 6500 Please refer to Figures 2a, 2b, and 2c for this section. Clients test1 through test8 were attached to the wiring closet switches in order to test network recovery times as different failures were induced. Each distribution-layer switch has a single connection into the core, but each building block as a whole has redundant connectivity into the core. This campus design features redundant Layer 3 switches in the distribution layer. For building block “west” the redundant switches are dwt65 (distribution west top 6500) and dwb65 (distribution west bottom 6500). Each distribution switch has a two-port Gigabit EtherChannel connection to a Layer 3 switch in the core. The HSRP track is configured on dwt65 and dwb65 so that fast HSRP recovery will take place if the routed link to the core is broken. There is one important caveat with this design: because each distribution-layer switch has a single path into the core, a redundant routed path must be provided in case this fails. If the physical link breaks, then HSRP will handle the recovery. However, in the event of some logical failure of the MSFC within the core switch, HSRP will not be triggered. An example of this is when the MSFC in the core router is reloaded. Eliminate the passive interface command on two of the wiring closet VLAN interfaces to provide two routed backup paths between the distribution-layer switches.
Table 3 Results with Small Routing Table
Test
Wiring Closet Uplink
Failure
Time of Interruption
Recovery Mechanism
Fail Restore Distribution Switch Fail Restore Core Switch Fail Restore
4s 1s
HSRP HSRP
4s 1s
HSRP HSRP
4s 14s
HSRP (See note below)
Note: When power is restored to the core switch, the links out to the distribution layer come up before EIGRP has stabilized. Therefore, HSRP on the distribution switch preempts the primary gateway router function a few seconds before EIGRP on the distribution switch has a routing table built; hence the 14-second interruption as noted. The preempt delay command-line option refers to delay upon powerup of the HSRP switch itself, and does not affect this value.
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 8 of 26
Table 4 Results with Large Routing Table
Test
Wiring Closet Uplink
Failure
Time of Interruption
Recovery Mechanism
Fail Restore Distribution Switch Fail Restore Core Switch Fail Restore
4s 1s
HSRP HSRP
4s 1s
HSRP HSRP
4s 30s
HSRP (See note below)
Note: When power is restored to the core switch, the links out to the distribution layer come up before EIGRP has stabilized. Therefore, HSRP on the distribution switch preempts the primary gateway router function a few seconds before EIGRP on the distribution switch has a routing table built; hence the 30-second interruption as noted. The preempt delay command-line option refers to delay upon powerup of the switch running HSRP in the distribution layer, and is not effective when the core switch is powered up.
Configuration of MSFC: Client-Side Interface on HSRP Primary Distribution Switch
interface Vlan10 ip address 10.10.0.81 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 (set eigrp hello timer 1 second) ip hold-time eigrp 1 3 (set eigrp hold timer 3 seconds) standby 10 timers 1 3 (set HSRP hello timer 1 hold timer 3 seconds) (convention - HSRP group number 10 matches VLAN number) standby 10 priority 200 preempt delay 60 (this is the primary gateway router for subnet 10) (preempt delay 60 seconds allows EIGRP to stabilize before HSRP switches back upon power recovery) standby 10 ip 10.10.0.200 (10.10.0.200 is the HSRP gateway router address) standby 10 track Vlan20 150 (if you lose the link to the backbone, drop priority by 150 to initiate HSRP recovery)
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 9 of 26
Configuration of MSFC: Client-Side Interface on HSRP Secondary Distribution Switch
interface Vlan10 ip address 10.10.0.82 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 standby 10 timers 1 3 standby 10 priority 100 preempt delay 60 (HSRP secondary or backup gateway router for subnet 10) standby 10 ip 10.10.0.200 standby 10 track Vlan23 50
Additional MSFC Configuration—Any Routed Interface to Backbone
interface Vlan21 ip address 10.21.0.81 255.255.0.0 no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3
Additional MSFC Configuration—Interface on Management VLAN
interface Vlan99 ip address 172.26.196.81 255.255.254.0 no ip directed-broadcast
Additional MSFC Configuration—EIGRP with Passive Interfaces to Wiring Closets
router eigrp 1 passive-interface Vlan10 passive-interface Vlan11 (no passive interface on VLAN 12 or VLAN 13) (two VLANs are kept as redundant routed paths) passive-interface Vlan99 network 10.0.0.0
3. Layer 3 Distribution with Dual-Path Layer 3 Core—Catalyst 6500 Please refer to Figures 3a, 3b, and 3c for this section. Clients test1 through test8 were attached to the wiring closet switches in order to test network recovery times as different failures were induced. This design has the highest level of redundancy and the highest link capacity into the core. This campus design features redundant Layer 3 switches in the distribution layer. For building block “west” the redundant switches are dwt65 (distribution west top 6500) and dwb65 (distribution west bottom 6500). Each distribution switch has redundant two-port Gigabit EtherChannel connections to both Layer 3 switches in the core. The HSRP track is configured on dwt65 and dwb65 so that fast HSRP recovery will take place if both routed links to the core are broken.
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 10 of 26
Table 5 Testing with the Standard Building Block—Small Routing Table
Test
Wiring Closet Uplink
Failure
Time of Interruption
Recovery Mechanism
Fail Restore Distribution Switch Fail Restore Core Switch Fail Restore
3s 0s
HSRP HSRP
3s 0s
HSRP HSRP
0s 6s
Dual-path redundancy EIGRP
Table 6 Testing with the Standard Building Block—Large Routing Table
Test
Wiring Closet Uplink
Failure
Time of Interruption
Recovery Mechanism
Fail Restore Distribution Switch Fail Restore Core Switch Fail Restore
3s 0s
HSRP HSRP
3s 0s
HSRP HSRP
0s 20s
Dual-path redundancy EIGRP
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 11 of 26
Table 7 Testing with the Standard Building Block—Small Routing Table
Test
Wiring Closet Uplink
Failure
Time of Interruption
Recovery Mechanism
Fail Restore Distribution Switch Fail
2s 0s
UplinkFast UplinkFast
3s
UplinkFast/HSRP
Restore Core Switch Fail Restore VLAN Backup Trunk Fail Restore
15s
(See note below)
3s 0s
Dual-path redundancy Dual-path redundancy
31s 28s
Spanning tree Spanning tree
Note: Restoration of the distribution-layer switch causes a 15-second outage. This is a result of the fact that the Layer 2 and Layer 3 path determination functions on the switch recover at slightly different moments. When the routing table is increased to 3000 routes in the next test, this anomaly is eliminated.
Table 8 Testing with the VLAN Building Block—Large Routing Table
Test
Wiring Closet Uplink
Failure
Time of Interruption
Recovery Mechanism
Fail Restore Distribution Switch Fail
2s 0s
UplinkFast UplinkFast
3s
UplinkFast/HSRP
Restore Core Switch Fail Restore VLAN Backup Trunk Fail Restore
5s
(See note below)
3s 0s
Dual-path redundancy Dual-path redundancy
31s 28s
Spanning tree Spanning tree
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 12 of 26
Note: When the distribution switch is restored, a five-second outage results because the Layer 2 topology stabilizes at a slightly different moment than the Layer 3 topology. With the large routing table this difference is reduced from the previous test.
Configuration of MSFC: Client-Side Interface on Distribution Switch—HSRP Primary
interface Vlan10 ip address 10.10.0.81 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 (set eigrp hello timer 1 second) ip hold-time eigrp 1 3 (set eigrp hold timer 3 seconds) standby 10 timers 1 3 (set HSRP hello timer 1 hold timer 3 seconds) (convention - HSRP group number 10 matches VLAN number) standby 10 priority 200 preempt delay 60 (this is the primary gateway router for subnet 10) (preempt delay 60 seconds allows EIGRP to stabilize before HSRP switches back upon power recovery) standby 10 ip 10.10.0.200 (10.10.0.200 is the HSRP gateway router address) standby 10 track Vlan20 75 standby 10 track Vlan21 75 (if you lose both links to the backbone, drop priority by 150 to initiate HSRP recovery)
Configuration of MSFC: Client-Side Interface on Distribution Switch—HSRP Secondary
interface Vlan10 ip address 10.10.0.82 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 standby 10 timers 1 3 standby 10 priority 100 preempt delay 60 (HSRP secondary or backup gateway router for subnet 10) standby 10 ip 10.10.0.200 standby 10 track Vlan22 25 standby 10 track Vlan23 25 (track both links to the backbone)
Additional MSFC Configuration—Any Routed Interface to Backbone
interface Vlan21 ip address 10.21.0.81 255.255.0.0 no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3
Additional MSFC Configuration—Interface on Management VLAN
interface Vlan99 ip address 172.26.196.81 255.255.254.0 no ip directed-broadcast
Additional MSFC Configuration—EIGRP with Passive Interfaces to Wiring Closets
router eigrp 1 passive-interface passive-interface passive-interface passive-interface passive-interface network 10.0.0.0
Vlan10 Vlan11 Vlan12 Vlan13 Vlan99
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 13 of 26
4. Layer 3 Distribution with Layer 3 Core—Catalyst 8540 Please refer to Figures A1a, A1b, and A1c for this section. Clients target1 through target6 were attached to the wiring closet switches in order to test network recovery times as different failures were induced. Each distribution-layer switch has a single connection into the core, but each building block as a whole has redundant connectivity into the core. This campus design features redundant Layer 3 switches in the distribution layer. For building block “left” the redundant switches are dlt85 (distribution left top 8540) and dlb85 (distribution left bottom 8540). Each distribution switch has a two-port Gigabit EtherChannel connection to a Layer 3 switch in the core. The HSRP track is configured on dlt85 and dlb85 so that fast HSRP recovery will take place if the routed link to the core is broken.
Table 9 Results with Small Routing Table
Test
Wiring Closet Uplink
Failure
Time of Interruption
Recovery Mechanism
Fail Restore Distribution Switch Fail Restore Core Switch Fail Restore
4s 1s
HSRP HSRP
3s 1s
HSRP HSRP
6s 3s
EIGRP HSRP
Table 10 Results with Large Routing Table
Test
Wiring Closet Uplink
Failure
Time of Interruption
Recovery Mechanism
Fail Restore Distribution Switch Fail Restore Core Switch Fail Restore
4s 1s
HSRP HSRP
3s 2s
HSRP HSRP
7s 3s
EIGRP HSRP
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 14 of 26
Configuration Details of HSRP Primary Distribution Router dlt85
interface Port-channel1 ip address 10.70.0.53 255.255.0.0 no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 hold-queue 300 in ! interface Port-channel2 ip address 10.71.0.53 255.255.0.0 no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 hold-queue 300 in ! interface GigabitEthernet0/0/1 no ip address no ip directed-broadcast channel-group 1 ! interface GigabitEthernet1/0/0 ip address 10.60.0.53 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 standby timers 1 3 standby priority 200 standby preempt standby ip 10.60.0.200 standby track Port-channel1 150 ! interface GigabitEthernet1/0/1 no ip address no ip directed-broadcast channel-group 1 ! interface GigabitEthernet2/0/0 ip address 10.61.0.53 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 standby timers 1 3 standby priority 200 standby preempt standby ip 10.61.0.200 standby track Port-channel1 150 ! interface GigabitEthernet2/0/1 no ip address no ip directed-broadcast channel-group 2 ! interface GigabitEthernet3/0/0 ip address 10.62.0.53 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 15 of 26
standby standby standby standby standby
timers 1 3 priority 200 preempt ip 10.62.0.200 track Port-channel1 150
! interface GigabitEthernet3/0/1 no ip address no ip directed-broadcast channel-group 2 ! interface Ethernet0 (management interface) ip address 172.26.196.53 255.255.254.0 no ip directed-broadcast ! ……… ! router eigrp 1 passive-interface GigabitEthernet0/0/0 passive-interface GigabitEthernet1/0/0 passive-interface GigabitEthernet2/0/0 passive-interface GigabitEthernet3/0/0 network 10.0.0.0 ! ……… end
5. Layer 3 Distribution with Dual-Path Layer 3 Core—Catalyst 8540 Please refer to Figures A2a, A2b, and A2c for this section. Clients target1 through target6 were attached to the wiring closet switches in order to test network recovery times as different failures were induced. The dual-path design has the highest level of redundancy and the highest link capacity into the core. This campus design features redundant Layer 3 switches in the distribution layer. For building block “left” the redundant switches are dlt85 (distribution left top 8540) and dlb85 (distribution left bottom 8540). Each distribution switch has redundant two-port Gigabit EtherChannel connections to both Layer 3 switches in the core. The HSRP track is configured on dlt85 and dlb85 so that fast HSRP recovery will take place if both routed links to the core are broken.
Table 11 Results with Small Routing Table
Test
Wiring Closet Uplink
Failure
Time of Interruption
Recovery Mechanism
Fail Restore Distribution Switch Fail Restore Core Switch Fail Restore
3s 0s
HSRP HSRP
3s 0s
HSRP HSRP
3s 1s
EIGRP EIGRP
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 16 of 26
Table 12 Results with Large Routing Table
Test
Wiring Closet Uplink
Failure
Time of Interruption
Recovery Mechanism
Fail Restore Distribution Switch Fail Restore Core Switch Fail Restore
3s 0s
HSRP HSRP
3s 11s
HSRP (See note below)
10s 1s
EIGRP EIGRP
Note: Restoring the distribution switch results in an outage between the time that HSRP switches and the EIGRP routing table is built. This can be corrected by adding a “preempt delay” as in the following command:
standby 10 priority 100 preempt delay 60
Configuration details of HSRP primary distribution router dlt85
interface Port-channel1 ip address 10.70.0.53 255.255.0.0 no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 hold-queue 300 in ! interface Port-channel2 ip address 10.71.0.53 255.255.0.0 no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 hold-queue 300 in ! interface GigabitEthernet0/0/1 no ip address no ip directed-broadcast channel-group 1 ! interface GigabitEthernet1/0/0 ip address 10.60.0.53 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 standby timers 1 3 standby priority 200 standby preempt standby ip 10.60.0.200 standby track Port-channel1 75 standby track Port-channel2 75 !
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 17 of 26
interface GigabitEthernet1/0/1 no ip address no ip directed-broadcast channel-group 1 ! interface GigabitEthernet2/0/0 ip address 10.61.0.53 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 standby timers 1 3 standby priority 200 standby preempt standby ip 10.61.0.200 standby track Port-channel1 75 standby track Port-channel2 75 ! interface GigabitEthernet2/0/1 no ip address no ip directed-broadcast channel-group 2 ! interface GigabitEthernet3/0/0 ip address 10.62.0.53 255.255.0.0 no ip redirects no ip directed-broadcast ip hello-interval eigrp 1 1 ip hold-time eigrp 1 3 standby timers 1 3 standby priority 200 standby preempt standby ip 10.62.0.200 standby track Port-channel1 75 standby track Port-channel2 75 ! interface GigabitEthernet3/0/1 no ip address no ip directed-broadcast channel-group 2 ! interface Ethernet0 ip address 172.26.196.53 255.255.254.0 no ip directed-broadcast ! ……… ! router eigrp 1 passive-interface GigabitEthernet0/0/0 passive-interface GigabitEthernet1/0/0 passive-interface GigabitEthernet2/0/0 passive-interface GigabitEthernet3/0/0 network 10.0.0.0 ! ……… end
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 18 of 26
1A: Dual-Path Layer 2 Core—Layout, Catalyst 6500
aw3 West Block Standard Test1 aw4 dwt65 ct65 det65 East Block VLAN
ae3
ae4
Test4
Test2
aw5
ae5
Test5
Test3
aw6 dwb65 cb65 deb65
ae6
Test6
Access
Distribution
Core
Distribution
Access
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
1B: Dual-Path Layer 2 Core—Logical, Catalyst 6500
aw3 10.10.0.101 10.10.0.0 172.26.196.11 West Block Standard East Block VLAN
ae3 10.50.0.101 10.5x.0.0 172.26.196.21
aw4 Test1 10.11.0.101 10.11.0.0 172.26.196.12
dwt65
ct65
det65
10.x.0.25 10.x.0.29 10.x.0.200 Pri 10.31.0.0 10.x.0.200 Pri 172.26.196.25 172.26.196.27 172.26.196.29
Test4 10.51.0.101 10.5x.0.0 172.26.196.22
ae4
aw5 Test2 10.12.0.101 10.12.0.0 172.26.196.13 10.x.0.26 10.x.0.30 10.32.0.0 10.x.0.200 Sec 10.x.0.200 Sec 172.26.196.28 172.26.196.26 172.26.196.30 dwb65 cb65 deb65
ae5 10.5x.0.0 172.26.196.23
Test5 10.52.0.101
aw6 Test3 10.13.0.101 10.13.0.0 172.26.196.14 Access
Test6 10.53.0.101 10.5x.0.0 172.26.196.24 Access
ae6
Distribution
Core
Distribution
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 19 of 26
1C: Dual-Path Layer 2 Core—Backbone, Catalyst 6500
West Block dwt65 VLAN 31 ct65 10.31.0.0
East Block det65 VLAN 31
VLAN 32
VLAN 32
VLAN 31 10.32.0.0 VLAN 32 dwb65 cb65
VLAN 31
VLAN 32 deb65
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
2A: Layer 3 Core—Layout, Catalyst 6500
aw3 West Block Standard Test1 aw4 dwt65 ct65 det65 East Block VLAN
ae3
ae4
Test4
Test2
aw5
ae5
Test5
Test3
aw6 dwb65 cb65 deb65
ae6
Test6
Access
Distribution
Core
Distribution
Access
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 20 of 26
2B: Layer 3 Core—Logical, Catalyst 6500
aw3 10.10.0.101 10.10.0.0 172.26.196.11 West Block Standard East Block VLAN
ae3 10.50.0.101 10.5x.0.0 172.26.196.21
aw4 Test1 10.11.0.101 10.11.0.0 172.26.196.12
dwt65
ct65
det65
10.x.0.25 10.x.0.29 10.x.0.200 Pri 10.x.0.27 10.x.0.200 Pri 172.26.196.25 172.26.196.27 172.26.196.29
Test4 10.51.0.101 10.5x.0.0 172.26.196.22
ae4
aw5 Test2 10.12.0.101 10.12.0.0 172.26.196.13 10.x.0.26 10.x.0.30 10.x.0.28 10.x.0.200 Sec 10.x.0.200 Sec 172.26.196.28 172.26.196.26 172.26.196.30 dwb65 cb65 deb65
Test5 10.52.0.101 10.5x.0.0 172.26.196.23
ae5
aw6 Test3 10.13.0.101 10.13.0.0 172.26.196.14 Access
ae6 10.5x.0.0 172.26.196.24 Access
Test6 10.53.0.101
Distribution
Core
Distribution
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
2C: Layer 3 Core—Backbone, Catalyst 6500
West Block dwt65 10.20.0.0 VLAN 20 VLAN 20 VLAN 30 VLAN 40 ct65
East Block det65 10.40.0.0 VLAN 40
10.30.0.0
VLAN 30 10.23.0.0 VLAN 23 dwb65 VLAN 23 cb65 VLAN 43 10.43.0.0 VLAN 43 deb65
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 21 of 26
3A: Dual-Path Layer 3 Core—Layout, Catalyst 6500
aw3 West Block Standard Test1 aw4 dwt65 ct65 det65 East Block VLAN
ae3
ae4
Test4
Test2
aw5
ae5
Test5
Test3
aw6 dwb65 cb65 deb65
ae6
Test6
Access
Distribution
Core
Distribution
Access
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
3B: Dual-Path Layer 3 Core—Logical, Catalyst 6500
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 22 of 26
3C: Dual-Path Layer 3 Core—Backbone, Catalyst 6500
West Block dwt65 VLAN 20 VLAN 21 10.21.0.0 VLAN 30 10.20.0.0 VLAN 20 VLAN 22 ct65 VLAN 40 VLAN 41
East Block 10.40.0.0 VLAN 40
det65
VLAN 42 10.42.0.0
10.30.0.0 10.22.0.0 VLAN 30 VLAN 22 10.23.0.0 VLAN 23 dwb65 Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
A1A: Layer 3 Core—Layout, Catalyst 8540
10.41.0.0 VLAN 41
VLAN 21 VLAN 23 cb65
VLAN 42 VLAN 43
10.43.0.0 VLAN 43 deb65
Left Block
Right Block
al4 dlt85 ct85 drt85
ar4
Target1
al5
ar5
Target3
Target2
al6 dlb85 Access Distribution cb85 Core drb85
ar6
Target4
Distribution
Access
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 23 of 26
A1B: Layer 3 Core—Logical, Catalyst 8540
Left Block
Right Block
al4 10.60.0.101 10.60.0.0 172.26.196.50 dlt85 ct65 drt85 10.x.0.53 10.x.0.59 10.x.0.200 Pri 10.x.0.57 10.x.0.200 Pri 172.26.196.53 172.26.196.57 172.26.196.59
ae4 10.100.0.101 10.100.0.0 172.26.196.61
al5 Target1 10.61.0.101 10.61.0.0 172.26.196.51 10.x.0.54 10.x.0.60 10.x.0.58 10.x.0.200 Sec 10.x.0.200 Sec 172.26.196.58 172.26.196.54 172.26.196.60 dlb85 cb85 drb85
ae5 10.101.0.0 172.26.196.62
Target3 10.101.0.101
al6 Target2 10.62.0.101 10.62.0.0 172.26.196.52 Access
Target4 10.102.0.101 10.102.0.0 172.26.196.63 Access
ae6
Distribution
Core
Distribution
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
A1C: Layer 3 Core—Topology, Catalyst 8540
Left Block dlt85 10.70.0.0 Port-chan 1 int gig 0/0/1 int gig 1/0/1 Port-chan 1 int gig 0/0/0 int gig 1/0/0 Port-chan 5 int gig 9/0/0 int gig 9/0/1 10.80.0.0 Port-chan 3 int gig 0/0/1 int gig 1/0/1 ct85
Right Block drt85 10.90.0.0 Port-chan 1 int gig 0/0/0 int gig 1/0/0
Port-chan 5 int gig 9/0/0 int gig 9/0/1 10.73.0.0 Port-chan 2 int gig 2/0/1 int gig 3/0/1 Port-chan 2 int gig 2/0/0 int gig 3/0/0 Port-chan 4 int gig 2/0/1 int gig 3/0/1 10.93.0.0 Port-chan 2 int gig 2/0/0 int gig 3/0/0
dlb85
cb85
drb85
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 24 of 26
A2A: Dual-Path Layer 3 Core—Layout, Catalyst 8540
Left Block
Right Block
al4 dlt85 ct85 drt85
ar4
Target1
al5
ar5
Target3
Target2
al6 dlb85 cb85 drb85
ae6
Target4
Access
Distribution
Core
Distribution
Access
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
A2B: Dual-Path Layer 3 Core—Logical, Catalyst 8540
Left Block
Right Block
al4 10.60.0.101 10.60.0.0 172.26.196.50 dlt85 ct85 drt85 10.x.0.53 10.x.0.59 10.x.0.200 Pri 10.x.0.57 10.x.0.200 Pri 172.26.196.53 172.26.196.57 172.26.196.59
ar4 10.100.0.101 10.100.0.0 172.26.196.61
al5 Target1 10.61.0.101 10.61.0.0 172.26.196.51 10.x.0.54 10.x.0.60 10.x.0.58 10.x.0.200 Sec 10.x.0.200 Sec 172.26.196.58 172.26.196.54 172.26.196.60 dlb85 cb85 drb85
ar5 10.101.0.0 172.26.196.62
Target3 10.101.0.101
al6 Target2 10.62.0.101 10.62.0.0 172.26.196.52 Access
ar6 10.102.0.0 172.26.196.63 Access
Target4 10.102.0.101
Distribution
Core
Distribution
Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual
Public Copyright © 2000 Cisco Systems, Inc. All Rights Reserved. Page 25 of 26
A2C: Dual-Path Layer 3 Core—Backbone, Catalyst 8540
Left Block dlt85 Port 1 Port 2 10.71.0.0 Port 5 10.70.0.0 Port 1 Port 2 ct85 Port 3 Port 4
Right Block 10.90.0.0 VLAN 40
drt85
Port 2 10.92.0.0
10.80.0.0 10.72.0.0 Port 5 Port 1 10.73.0.0 Port 2 dlb85 Gigabit VLAN Trunk Gigabit Ethernet Gigabit EtherChannel—Dual Port 1 Port 2 cb85 Port 3 Port 4 10.93.0.0 Port 2 drb85
10.91.0.0 Port 1
Conclusion
Choose a deterministic, structured design model to achieve high availability in your enterprise network. Apply redundancy in the mission-critical parts of the network. Scale the enterprise network to the size required by choosing the appropriate building block model and combining with the right backbone model. For a better understanding of the different choices referred to in this paper refer to the paper Gigabit Campus Network Design—Principles and Architecture.
Geoff Haviland ([email protected])—Network Design Engineer.
Corporate Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 526-4100
European Headquarters Cisco Systems Europe s.a.r.l. Parc Evolic, Batiment L1/L2 16 Avenue du Quebec Villebon, BP 706 91961 Courtaboeuf Cedex France http://www-europe.cisco.com Tel: 33 1 69 18 61 00 Fax: 33 1 69 28 83 26
Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-7660 Fax: 408 527-0883
Asia Headquarters Nihon Cisco Systems K.K. Fuji Building, 9th Floor 3-2-3 Marunouchi Chiyoda-ku, Tokyo 100 Japan http://www.cisco.com Tel: 81 3 5219 6250 Fax: 81 3 5219 6001
Cisco Systems has more than 200 offices in the following countries. Addresses, phone numbers, and fax numbers are listed on the
C i s c o C o n n e c t i o n O n l i n e We b s i t e a t h t t p : / / w w w. c i s c o . c o m / o f f i c e s .
Argentina • Australia • Austria • Belgium • Brazil • Canada • Chile • China • Colombia • Costa Rica • Croatia • Czech Republic • Denmark • Dubai, UAE Finland • France • Germany • Greece • Hong Kong • Hungary • India • Indonesia • Ireland • Israel • Italy • Japan • Korea • Luxembourg • Malaysia Mexico • The Netherlands • New Zealand • Norway • Peru • Philippines • Poland • Portugal • Puerto Rico • Romania • Russia • Saudi Arabia • Singapore Slovakia • Slovenia • South Africa • Spain • Sweden • Switzerland • Taiwan • Thailand • Turkey • Ukraine • United Kingdom • United States • Venezuela
Copyright © 2000 Cisco Systems, Inc. All rights reserved. Printed in the USA. Catalyst, Cisco, Cisco IOS, Cisco Systems, and the Cisco Systems logo are registered trademarks of Cisco Systems, Inc. or its affiliates in the U.S. and certain other countries. All other trademarks mentioned in this document are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any of its resellers. (9912R) 02/00 BW5858
