Nexus 7000
The Cisco Nexus 7000 Series is a modular data center class series of switching systems designed for highly scalable end-to-end 10 Gigabit Ethernet networks. The Cisco Nexus 7000 Series is purpose built for the data center and has many unique features and capabilities designed specifically for such mission critical place in the network.
Cisco NX-OS, a state-of-the-art operating system, powers the Cisco Nexus 7000 Platform. Cisco NX-OS is built with modularity, resiliency, and serviceability at its foundation. Drawing on its Cisco IOS and Cisco SAN-OS heritage, Cisco NX-OS helps ensure continuous availability and sets the standard for mission-critical data center environments.
Lab Objectives
This instructor-led hands-on lab will introduce the participants to the OTV (Overlay Transport Virtualization) solution for the Nexus 7000. This innovative feature set simplifies Datacenter Interconnect designs, allowing multisite Data Center communication and transparent Layer 2
extension across multiple Data Center sites. OTV accomplishes this without the overhead introduced by MPLS or VPLS. By the end of the laboratory session the participant should be able to understand basic and advanced OTV functionality and configuration with the Nexus 7000.
General Disclaimer
The content of this Lab is based on a feature set not presently released by Cisco. Feature content, configuration commands and terminology are subjected to change at any point in time until official code availability. The Lab Hands-on session will be delivered using prerelease Early Field Trial code.
Lab Procedure
The Lab consists of 8 PODs. Each single POD represents a typical but simplified Nexus 7000 Data Center site where Nexus 7000 is used as an edge device attached to a layer 3 Core cloud. The core consists of a pair of Catalyst 6500s. Catalyst 6500s are not the typical DC core platform and are here only used for convenience to model a simplified L3 core network. A Nexus 5000 with an attached ESX server represent the access layer. The aggregation layer (on which all the configuration for this lab is performed) is built on Nexus 7000 10-slot chassis with dual supervisors, one 48-port GE Copper card (model N7KM148GT-12) and one 32-port 10GE fiber card (model N7K-M132XP-12) each. Nexus 7000 devices run a PRE-Release Early Field Trial version of the NX-OS 5.0(2). One student is assigned to each Pod. A student will be able to configure his own Nexus 7000 aggregation device. During the Lab procedure the students will go through the following steps: System Verification: Management VRF, Basic Connectivity, CLI Familiarization, Base configuration: Spanning Tree, LACP and OSPF. Configuring OTV to establish adjacencies with 2 Head-End remote sites. Verifying OTV environment. Testing the multisite connectivity and remote MAC learning.
Lab Topology and Access
In this multi-site Data-Center environment Each student will be able to configure a single Nexus 7000 device in a pre-assigned site. The goal of the lab is to establish L2 connectivity with remote end sites over a generic IP core leveraging the Nexus 7000 Overlay Transport technology.
Figure 1 –Multi-Site Data-Center Topology (Overview of all the PODs)
Each site has independent connectivity through the IP Core infrastructure, as described in Figure 2 .
Each POD has its own set of pre-assigned interfaces and IP addresses. The diagrams below represent topologies for odd pods (Pod1, Pod3, Pod5, Pod7) and even pod (Pod2, Pod4, Pod6, Pod8).
In the present LAB we leverage the Virtual Device Context feature to consolidate multiple nodes and reduce the required equipment.
N7K-Edge
Pod 1
Pod 2
Figure 5 - multi-VDC POD deployment
Based on your POD number, identify which of the topologies (odd or even) you will be working with and which IP addresses and interfaces you will be using. All access to your POD devices is via the ESX VMware server that is available via the Microsoft Remote Desktop (Microsoft Remote Desktop client can be found under start>accessories>communication). Credential to be used for the Remote desktop are defined in Table 1. PLEASE NOTE: Interfaces and IP addresses referenced throughout various steps of the guide will change based on your target topology (Odd or Even POD # Topology). CLI commands have x, y, z parameters indicating such variable configuration. Please refer to Figure 3 and Figure 4 to identify the right interfaces and IP addresses The Output of the show commands in this guide refers to POD 1 and so may slightly vary based on the POD # you are operating with.
1. Open the Microsoft Remote Desktop Client on your workstation and point your
machine to the Pod‟s VM instance as shown in Table 1.
POD Information Remote Desktop VM IP address Login/Password
Table 1 - POD Access Details
2. For your convenience you will find puTTY connections for all relevant devices on the Desktop. Double click on the connection of the Nexus 7000. Click YES to proceed if you get an SSH warning message.
Figure 6 - Remote Shared Desktop Environment with puTTY consoles
Step 1 System Verification
PLEASE NOTE: This section is not required to complete the OTV configuration. You can skip this step if you are already familiar with the Nexus 7000 hardware and software infrastructure. In this case jump to Step 2 (CLI Familiarization paragraph). During the entire duration of this lab we will be just logging into the management interface via ssh. However it is good to keep in mind that the Nexus 7000 requires console access to perform the initial configuration of the system. After performing the initial configuration, the system can be completely managed from the management interface. Let’s start by checking the system and its configuration.
N7K-1-pod1-S1# show module Mod --1 2 5 6 Mod --1 2 Ports ----48 32 0 0 Module-Type -------------------------------10/100/1000 Mbps Ethernet Module 10 Gbps Ethernet Module Supervisor module-1X Supervisor module-1X Hw -----1.0 1.3 Model -----------------N7K-M148GT-11 N7K-M132XP-12 N7K-SUP1 N7K-SUP1 Status -----------ok ok ha-standby active *
Let’s check now the software the system is running.
N7K-1-pod1-S1# show version Cisco Nexus Operating System (NX-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2010, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php Software BIOS: loader: version 3.17.0 version N/A
NX-OS Version kickstart: version 5.0(2) system: version 5.0(2) BIOS compile time: 03/23/08 kickstart image file is: bootflash:/n7000-s1-kickstart.5.0.2.S10.gbin kickstart compile time: 12/25/2020 12:00:00 [02/25/2010 03:44:41] system image file is: bootflash:/n7000-s1-dk9.5.0.2.S10.gbin Images Location system compile time: 2/7/2010 3:00:00 [02/25/2010 04:34:08] Hardware cisco Nexus7000 C7010 (10 Slot) Chassis ("Supervisor module-1X") Intel(R) Xeon(R) CPU with 4129620 kB of memory. Processor Board ID JAB123501Z7 Device name: N7K-1 bootflash: 2000880 kB slot0: 0 kB (expansion flash) Storage Devices
CPU
Kernel uptime is 0 day(s), 23 hour(s), 39 minute(s), 59 second(s) Last reset at 185087 usecs after Tue Dec 2 04:59:22 2008
Reason: Reset Requested by CLI command reload System version: 4.1(1.66) Service: plugin Core Plugin, Ethernet Plugin N7K-1-pod1-S1# Active Plug-in
1. NX-OS is composed by two images: a kickstart image that contains the Linux Kernel and a
system image that contains the NX-OS software components. They both show up in the configuration.
2. In a future release we will be adding other plug-ins, like the “Storage” plug-in for FCoE.
Let’s now take a look at the running configuration.
N7K-1-pod1-S1# show running-config version 5.0(2) <omitted config> vrf context management vlan 1 <omitted interface config> interface Ethernet2/1 interface Ethernet2/2 <omitted interface config> interface Ethernet2/16 These are the interfaces available to your Pod (Virtual Device Context)
This is the configuration of the first Pod. As explained earlier each Pod runs within a Virtual Device Context (VDC). By using the VDC feature, we can segment the physical Nexus 7000 into multiple logical switches, each of which runs in a separate memory space and only has visibility into the hardware resources that it owns, providing total isolation between the VDCs.
3.
One of the features of “show running-config” in NX-OS consists in the ability to not only look at the running-config but to also at the default values, which do not appear in the base config. The keyword to use is “all”.
N7K-1-pod1-S1# show running-config all | section mgmt0 interface mgmt0 no description speed auto duplex auto snmp trap link-status no shutdown cdp enable ip redirects ip address 192.168.1.17/16 ip port-unreachable ipv6 redirects ip arp gratuitous update ip arp gratuitous request
Management VRF and Basic Connectivity
The management interface is always part of the management VRF. The management interface “mgmt0” is the only interface allowed to be part of this VRF.
The Management VRF provides total isolation of management traffic from the rest of the traffic flowing through the box. In this step we will: - Verify that only the mgmt0 interface is part of the management VRF
- Verify that no other interface can be part of the management VRF - Verify that the default gateway is reachable only using the management VRF
N7K-1-pod1-S1# show vrf VRF-Name default management VRF-ID 1 2 State Up Up Reason ---
The management VRF interface is part of the default configuration and the management interface “mgmt0” is the only interface that can be made member of this VRF. Let’s verify it.
4. N7K-1-pod1-S1# conf t N7K-1-pod1-S1(config)# interface ethernet 2/y N7K-1-pod1-S1(config-if)# vrf member management % VRF management is reserved only for mgmt0 N7K-1-pod1-S1(config-if)# show int mgmt0 FastEthernet? GigabitEthernet?... no, just “ethernet” interfaces Use Ethernet 2/10 for the Odd POD #s (i.e. POD #1, 3, 5, 7) and Ethernet 2/26 for Even POD #s (i.e. POD 2, 4, 6, 8 )
mgmt0 is up DCE oper mode is auto Hardware: GigabitEthernet, address: 0022.5577.f8a8 (bia 0022.5577.f8a8) Internet Address is 192.168.1.17/16 MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA full-duplex, 1000 Mb/s Auto-Negotiation is turned on EtherType is 0x0000 1 minute input rate 72 bits/sec, 0 packets/sec 1 minute output rate 24 bits/sec, 0 packets/sec Rx 353 input packets 0 unicast packets 265 multicast packets 88 broadcast packets 51632 bytes Tx 92 output packets 0 unicast packets 91 multicast packets
Try to reach the out-of-band management network’s default gateway with a ping.
N7K-1-pod1-S1(config-if)# ping 192.168.0.1 PING 192.168.0.1 (192.168.0.1): 56 ping: sendto 192.168.0.1 64 chars, Request 0 timed out ping: sendto 192.168.0.1 64 chars, Request 1 timed out ping: sendto 192.168.0.1 64 chars, Request 2 timed out ping: sendto 192.168.0.1 64 chars, Request 3 timed out ping: sendto 192.168.0.1 64 chars, Request 4 timed out data bytes No route to host No route to host No route to host No route to host No route to host
Step 2 CLI Familiarization
NX-OS CLI is very IOS-like as you will notice when configuring the system. Also NX-OS implements a hierarchy independent CLI, so that any command can be issued from any CLI context.
PLEASE NOTE: This section is not required to complete the OTV configuration so you can skip this step if you are already familiar with the Nexus 7000 CLI capabilities. In this case jump to Step 3 (Spanning Tree configuration paragraph). In this step we will: - Verify the CLI hierarchy independence by issuing a ping from different CLI contexts - Verify the CLI piping functionality
N7K-1-pod1-S1# conf t N7K-1-pod1-S1(config)# ping ? *** No matches in current mode, matching in (exec) mode *** <CR> A.B.C.D or Hostname IP address of remote system WORD Enter Hostname multicast Multicast ping N7K-1-pod1-S1(config)# ping 192.168.0.1 vrf management PING 192.168.0.1 (192.168.0.1): 56 data bytes 64 bytes from 192.168.0.1: icmp_seq=0 ttl=63 time=4.257 64 bytes from 192.168.0.1: icmp_seq=1 ttl=63 time=0.714 64 bytes from 192.168.0.1: icmp_seq=2 ttl=63 time=0.562 64 bytes from 192.168.0.1: icmp_seq=3 ttl=63 time=0.581 64 bytes from 192.168.0.1: icmp_seq=4 ttl=63 time=0.568 ms ms ms ms ms
Hierarchy Independent CLI
--- 192.168.0.1 ping statistics --5 packets transmitted, 5 packets received, 0.00% packet loss round-trip min/avg/max = 0.562/1.336/4.257 ms Use Ethernet 2/10 for the Odd POD #s N7K-1-pod1-S1(config)# int e2/y (i.e. POD #1, 3, 5, 7) and Ethernet 2/26 for Even POD #s (i.e. POD 2, 4, 6, 8 ) N7K-1-pod1-S1(config-if)# ping ? *** No matches in current mode, matching in (exec) mode *** <CR> A.B.C.D or Hostname WORD multicast IP address of remote system Enter Hostname Multicast ping CLI Hierarchy Independent
N7K-1-pod1-S1(config-if)# ping 192.168.0.1 vrf management PING 192.168.0.1 (192.168.0.1): 56 data bytes 64 bytes from 192.168.0.1: icmp_seq=0 ttl=63 time=3.768 64 bytes from 192.168.0.1: icmp_seq=1 ttl=63 time=0.713 64 bytes from 192.168.0.1: icmp_seq=2 ttl=63 time=0.586 64 bytes from 192.168.0.1: icmp_seq=3 ttl=63 time=0.592 64 bytes from 192.168.0.1: icmp_seq=4 ttl=63 time=0.597 --- 192.168.0.1 ping statistics --ms ms ms ms ms
5 packets transmitted, 5 packets received, 0.00% packet loss round-trip min/avg/max = 0.586/1.251/3.768 ms
6. 7.
You can use the up-arrow and get the command history from the exec mode Any command can be issued from anywhere within the configuration
Multiple piping options are available lots of them derived from the Linux world.
N7K-1-pod1-S1# show running-config | ? cut diff Enhanced CLI Piping Print selected parts of lines. Show difference between current and previous invocation (creates temp files: remove them with 'diff-clean' command and dont use it on commands with big outputs, like 'show tech'!) Egrep - print lines matching a pattern Grep - print lines matching a pattern Display first lines Output in human format (if permanently set to xml, else it will turn on xml for next command) Display last lines Filter for paging Turn-off pagination for command output Use perl script to filter output Show lines that include the pattern as well as the subsequent lines that are more indented than matching line Stream Editor Stream Sorter Stream SCP (secure copy) Translate, squeeze, and/or delete characters Discard all but one of successive identical lines The shell that understands cli command Count words, lines, characters Output in xml format (according to .xsd definitions) Begin with the line that matches Count number of lines End with the line that matches Exclude lines that match Include lines that match
egrep grep head human last less no-more perl section sed sort sscp tr uniq vsh wc xml begin count end exclude include
N7K-1-pod1-S1# sh running-config | grep ? WORD count ignore-case invert-match line-exp line-number next prev word-exp Search for the expression Print a total count of matching lines only Ignore case difference when comparing strings Print only lines that contain no matches for <expr> Print only lines where the match is a whole line Print each match preceded by its line number Print <num> lines of context after every matching line Print <num> lines of context before every matching line Print only lines where the match is a complete word
The following command will grab the instance of a line with “mgmt0” and print the following 3 lines after that match.
N7K-1-pod1-S1# sh running-config | grep next 3 mgmt0 interface mgmt0 no snmp trap link-status ip address 192.168.1.17/16 N7K-1-pod1-S1# conf t N7K-1-pod1-S1(config)# int mgmt 0 N7K-1-pod1-S1(config-if)# [TAB] cdp end ipv6 control exit no description ip pop 8.
push shutdown snmp
this vrf where
The [TAB] completes the CLI command and shows the available keywords.
If you want to know the CLI context you are in use the “where” command.
N7K-1-pod1-S1(config-if)# where conf; interface mgmt0 admin@N7K-1-pod1-S1%default
Step 3 Spanning Tree
It is time to bring up the interfaces and configure the Spanning Tree Protocol. Rapid Spanning Tree Protocol (RSTP) is standardized in 802.1d (now, IEEE 802.1D-2004). Cisco's implementation of RSTP in both NX-OS and IOS provides a separate spanning tree instance for each active VLAN, which permits greater flexibility of Layer 2 topologies in conjunction with IEEE 802.1Q trunking. This implementation is also referred to as Rapid PerVLAN Spanning Tree (Rapid-PVST). Rapid-PVST is the default spanning tree mode for NX-OS, so it does not need to be explicitly enabled. Best practices dictate deterministic placement of the spanning tree root in the network. Particularly a network administrator should ensure that a root switch does not inadvertently end up on a small switch in the access layer creating a sub-optimal topology more prone to failures. Let’s first configure the VLANs in each data-center site
9. Each site will have 2 different VLANs, one local to the site and one to be extended on
the overlay to the remote data-center sites. VLANs are x0 and x00 where x identifies the POD# (i.e. 10 and 100 for POD 1, 20 and 200 for POD 2 and so on).
N7K-1-pod1-S1# conf t Enter configuration commands, one per line.
Now let’s bring up the interfaces facing the Access Layer (i.e. facing the Nexus 5000)
N7K-1-pod1-S1(config-if-range)# N7K-1-pod1-S1(config-if-range)# N7K-1-pod1-S1(config-if-range)# N7K-1-pod1-S1(config-if-range)# N7K-1-pod1-S1(config-if-range)# int ethernet 2/y switchport switchport mode trunk switchport trunk allowed vlan x0, x00 no shutdown
Interface numbers will change for odd and even POD #s. Check your Network topology to target the right interfaces
Now let’s create these VLANs on the attached Nexus 5000 access device. Double-click on the puTTY console shortcut for the Nexus 5000 on your shared desktop. Check the Network Diagram and based on your POD find out which VLAN you need to create and add those on the interfaces connecting to your Nexus 7000 (i.e. 10 and 100 for POD 1, 20 and 200 for POD 2 and so on).
N5K-1# conf t N5K-1(config)# vlan x0, x00 N5K-1(config-vlan)# no shut
Now configure interfaces toward the Nexus 7000
N5K-1(config)# int N5K-1(config-if)# N5K-1(config-if)# N5K-1(config-if)# N5K-1(config-if)# ethernet 1/z switchport switchport mode trunk switchport trunk allowed vlan x0, x00 no shutdown
Interface numbers will change for odd and even POD #s. Check your Network topology to target the right interfaces
And enable VLAN to be extended on the interface toward the End Host/Server. When you configure the port as STP type edge (i.e. equivalent of portfast) an appropriate informational warning message is printed. Make Sure you extend the right VLAN to the HOST (VLAN 100 for Pod 1, VLAN 200 for POD2, VLAN 300 for POD3 and so on)
N5K-1(config)# int ethernet 1/3 N5K-1(config-if)# switchport N5K-1(config-if)# switchport mode access N5K-1(config-if)# switchport access vlan x00 N5K-1(config-if)# spanning-tree port type edge N5K-1(config-if)# no shut Warning: Edge port type (portfast) should only be enabled on ports connected to a single host. Connecting hubs, concentrators, switches, bridges, etc... to this interface when edge port type (portfast) is enabled, can cause temporary bridging loops. Use with CAUTION
Check the spanning-tree from both the Nexus 7000 …
N7K-1-pod1-S1# show spanning-tree vlan x00 VLAN0100 Spanning tree enabled protocol rstp Root ID Priority 4196 Address 0022.5579.c442 This bridge is the root Hello Time 2 sec Max Age 20 sec Bridge ID Priority Address Hello Time
Forward Delay 15 sec
4196 (priority 4096 sys-id-ext 100) 0022.5579.c442 2 sec Max Age 20 sec Forward Delay 15 sec
Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- ------------------------------Eth2/9 Desg FWD 2 128.265 P2p N7K-1-pod1-S1(config-if-range)# show spanning-tree vlan x0 VLAN0010 Spanning tree enabled protocol rstp Root ID Priority 4106 Address 0022.5579.c442 This bridge is the root Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Bridge ID Priority Address Hello Time 4106 (priority 4096 sys-id-ext 10) 0022.5579.c442 2 sec Max Age 20 sec Forward Delay 15 sec
Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -----------------------------Eth2/9 Desg FWD 2 128.265 P2p
… and the and the Nexus 5000
N5K-1# show spanning-tree vlan x00 VLAN0100 Spanning tree enabled protocol rstp Root ID Priority 4196 Address 0022.5579.c442 Cost 2 Port 129 (Ethernet1/1) Hello Time 2 sec Max Age 20 sec Bridge ID Priority Address Hello Time Role ---Root Desg
Forward Delay 15 sec
32868 (priority 32768 sys-id-ext 100) 000d.eca4.04fc 2 sec Max Age 20 sec Forward Delay 15 sec Cost --------2 2 Prio.Nbr -------128.129 128.131 Type ------------------------------P2p Edge P2p
Step 4 Configuring OTV to connect edge devices to remote end-sites
OTV provides Layer 2 connectivity between remote network sites. OTV uses MAC addressbased routing and IP-encapsulated forwarding across a Layer 3 network to provide support for applications that require Layer 2 adjacency, such as clusters and vmotion. You deploy OTV on the edge devices in each site. OTV requires no other changes to the sites or the core network. OTV avoids the addition of multiple routing tables to every device in the network that other methods, such as Multiprotocol Label Switching (MPLS), require.
Figure 7 - OTV Packet Flow
The following terminology is used for OTV throughout this document: Site: A Layer 2 network that may be single-homed or multihomed to the core network and the OTV overlay network. Layer 2 connectivity between sites is provided by edge devices that operate in an overlay network. Layer 2 sites are physically separated from each other by the core IP network. Core Network: The customer backbone network that connects Layer 2 sites over IP. This network can be customer managed, provided by a service provider, or a mix of both. OTV is transparent to the core network because OTV flows are treated as regular IP flows. Edge Device: A Layer 2 switch that performs OTV functions. An edge device performs typical Layer 2 learning and forwarding on the site-facing interfaces (internal interfaces) and performs IP-based virtualization on the core-facing interfaces. The edge device can be collocated in a device that performs Layer 3 routing on other
ports. OTV functionality only occurs in an edge device. Internal Interface: The Layer 2 interface on the edge device that connects to sitebased switches or site-based routers. The internal interface is a Layer 2 interface regardless of whether the internal interface connects to a switch or a router. Join Interface: The interface facing the core network. The name implies that the edge device joins an overlay network through this interface. The IP address of this interface is used to advertise reachability of a MAC address present in this site.
Figure 8 - OTV Terminology (1 of 2)
MAC Routing: MAC routing associates the destination MAC address of the Layer 2 traffic with an edge device IP address. The MAC to IP association is advertised to the edge devices through an overlay routing protocol. In MAC routing, MAC addresses are reachable through an IP next hop. Layer 2 traffic destined to a MAC address will be encapsulated in an IP packet based on the MAC to IP mapping in the MAC routing table. Overlay Interface: A logical multi-access multicast-capable interface. The overlay interface encapsulates Layer 2 frames in IP unicast or multicast headers. The overlay interface is connected to the core via one or more physical interfaces. You assign IP addresses from the core network address space to the physical interfaces that are associated with the overlay interface. Overlay Network: A logical network that interconnects remote sites for MAC routing of Layer 2 traffic. The overlay network uses either multicast routing in the core network or an overlay server to build an OTV routing information base (ORIB). The ORIB associates destination MAC addresses with remote edge device IP addresses. Authoritative Edge Device: An edge device that forwards Layer 2 frames into and out of a site over the overlay interface. For the first release of OTV, there is only one
authoritative edge device for all MAC unicast and multicast addresses per VLAN. Each VLAN can be assigned to a different authoritative edge device.
Figure 9- OTV Terminology (2 of 2)
In this section you will: Select the Join interface and establish OSPF connectivity with the Core. Enable OTV Configure the Overlay interface Join the Data-Center site to the Core via the join interface. Extend a VLAN across the overlay to connect the local site with the remote sites.
10. In this first step we will identify the interconnection to the core and configure OSPF for L3
connectivity. This interface will be assigned as the Join interface of the OTV Edge device Let’s now select the join interface on the Nexus 7000 edge device. Look at the topology diagram and based on your POD topology pick one of the 2 uplink interfaces connected to the Core. First, un-shut both connections to the core (i.e. selecting a range of interfaces):
Interface numbers will change for odd and even POD #s. Check your Network topology to target the right interfaces and determine „y‟ and „z‟ values. R S I R S I S I s WS-C6503 Gig2/1 WS-C6503 Gig2/2 N5K-C5020P-BF Eth1/1
N7K-1-pod1-S1(config)# int e 1/y-z N7K-1-pod1-S1(config-if-range)# no shut N7K-1-pod1-S1# sh cdp neighbors C6K-2 C6K-1 N5K-1(FLC12220548) Eth1/13 Eth1/14 Eth2/9 139 133 161
11. NX-OS is a fully modular operating system; most software modules don’t run unless
the correspondent service is enabled. We refer to these features that need to be specifically enabled as “conditional services”. Once the service is enabled, the CLI becomes visible and the feature can be used and configured.
Now let’s configure Layer 3 and OSPF Routing
N7K-1-pod1-S1(config)# conf t N7K-1-pod1-S1(config)# feature ospf N7K-1-pod1-S1(config)# router ospf 1
Enable OSPF as conditional service
Check the Network diagram to assign the right IP addresses to your POD. „X‟ indicated here matches the POD# (i.e. „1‟ for POD1, „2‟ for POD2 and so on). „y‟ and „z‟ will differ based on the interface IDs (check the right topology diagram)
N7K-1-pod1-S1(config)# interface loopback0 N7K-1-pod1-S1(config-if)# ip address 10.99.x.1/32 N7K-1-pod1-S1(config-if)# ip router ospf 1 area 0.0.0.0 N7K-1-pod1-S1(config)# interface Ethernet1/<first_uplink> N7K-1-pod1-S1(config-if)# ip address 10.x.y.1/30 N7K-1-pod1-S1(config-if)# ip ospf network point-to-point N7K-1-pod1-S1(config-if)# ip router ospf 1 area 0.0.0.0 N7K-1-pod1-S1(config-if)# ip igmp version 3 N7K-1-pod1-S1(config-if)# no shutdown N7K-1-pod1-S1(config)# interface Ethernet1/<second_uplink> N7K-1-pod1-S1(config-if)# ip address 10.x.z.1/30 N7K-1-pod1-S1(config-if)# ip ospf network point-to-point N7K-1-pod1-S1(config-if)# ip router ospf 1 area 0.0.0.0 N7K-1-pod1-S1(config-if)# ip igmp version 3 N7K-1-pod1-S1(config-if)# no shutdown
Interface numbers may change for odd and even POD #s. Check your Network topology to target right interfaces (i.e. Ethernet 1/13-14 or Ethernet 1/25-26)
Let’s check OSPF neighbors to see if the OSPF connectivity was successfully established.
N7K-1-pod1-S1# sh ip ospf neighbors OSPF Process ID 1 VRF default Total number of neighbors: 2 Neighbor ID Pri State 10.99.12.2 1 FULL/ 10.99.12.1 1 FULL/ -
Up Time Address 00:18:14 10.1.2.2 00:18:13 10.1.1.2
Interface Eth1/13 Eth1/14
12. In this second step we configure the OTV Overlay interface and join the Overlay transport
through the IP core. Let’s first enable the OTV feature set. Note: it may take few seconds to enable the required set of protocols, but this is a one-time operation.
N7K-1-pod1-S1(config)# conf t N7K-1-pod1-S1(config)# feature otv
We will use the pre-configured VLANs <X>00 as VLAN to extend and VLAN <X>0 as site VLAN. X is again the POD# (VLAN 100 and VLAN 10 for POD 1, VLAN 200 and VLAN 20 for POD 2 and so on). The site VLAN is the one used to communicate with other edge devices in the local site. In this case it will be not used as we only have a single edge device in the site, however for completeness we will configure it.
N7K-1-pod1-S1(config)# conf t N7K-1-pod1-S1(config)# otv site-vlan x0 VLANs will change based on the POD #s.
Now let’s specify for POD X the Overlay configuration, X is the POD# (Overlay 1 for POD 1, Overlay 2 for POD 2 and so on).
N7K-1-pod1-S1(config)# interface Overlay <X> N7K-1-pod1-S1(config-if-overlay)# N7K-1-pod1-S1(config-if-overlay)# otv control-group 239.<X>.1.1 otv data-group 239.<X>.2.0/28
Where X is again the POD# (1 for POD 1, 2 for POD 2 and so on). The group address is used for control plane related operations. Each edge device joins the group and sends control/protocol related packets to this group. This is used for discovery of other edgedevices. The data-group-range specifies a multicast group range that is used for multidestination traffic (doesn't hit CPU of remote edge devices). Now join the site you are working on with the next hop core device selecting the join interface. This command assigns an L3 interface as the core-facing interface for OTV. This interface is used for overlay operations such as discovering remote edge-devices, providing the source address for OTV encapsulated packets and the destination address for unicast traffic sent by remote edge-devices. Each POD has two ECMP interface connections to the core and one of these should be used as join interface. After you enter a command an informational message reminds you that IGMPv3 is required to be configured on the join interface. Message can be just ignored if IGMPv3 was already configured as instructed earlier in the guide.
N7K-1-pod1-S1(config-if-overlay)# otv join-interface Ethernet1/y OTV needs join interfaces to be configured for IGMP version 3 Join Interface (uplink interface to the Core) may change for odd and even POD #s. Check your Network topology to target right interfaces (i.e. Ethernet 1/13-14 or Ethernet 1/25-26)
Last let’s pick a VLAN to be extended. Multiple VLANs and a VLAN range could be extended however just for a practical demonstration we will extend a single VLAN across the core to reach the two remote Head-End Data-Centers.
N7K-1-pod1-S1(config-if-overlay)# otv extend-vlan ? , Multi range separator Range separator <1-3967,4048-4093> VLAN ID 1-4094 or range(s): 1-5, 10 or 2-5,7-19
Step 5 OTV verification and Monitoring Commands
In this step we will monitor and troubleshoot the OTV configuration. First of all let’s display the OTV overlay status and parameters for your POD (local site):
N7K-1-pod1-S1# show otv overlay <Pod_Number> OTV Overlay Information Overlay Interface Overlay1 VPN Name : VPN ID : State : IPv4 multicast group : IPv6 multicast group : Mcast data group range(s): External interface(s) : External IPv4 address : External IPv6 address : Encapsulation format : Site-vlan : Capability : Is Adjacency Server : Adj Server Configured : Prim/Sec Adj Svr(s) :
Overlay1 192 UP Overlay1-239.1.1.1 [None] 239.1.2.0/28 Ethernet1/13 10.1.2.1 0:: GRE/IPv4 10 Multicast-Reachable NO NO [None] / [None]
Also let’s verify what VLANs are being extended:
N7K-1-pod1-S1# show otv vlan OTV VLAN Configuration Information VLAN-ID VlanState Switchport/ Forward Count 100 UP 1/1
External Interface Ethernet1/13
Overlay Group Overlay1-239.1.1.1
Now let‟s check how many OTV edge devices are present on the site. Because this is a single-homed site, one node will be listed through this command. The „*‟ symbol next to the MAC address indicates the local node.
N7K-1-pod1-S1# show otv site all OTV Overlay Information Site-VLAN : 10
The authoritative device is the OTV node elected to forward traffic to/from the L3 core. Only one authoritative device will be elected in a site. The show command below returns a non empty output only on the authoritative OTV data-center edge node.
N7K-1-pod1-S1# show otv vlan authoritative OTV VLAN Configuration Information VLAN-ID VlanState Switchport/ Forward Count 100 UP 1/1
The MAC address table will report mac addresses of end-hosts and devices learnt on the VLAN. If no traffic was ever sent across the overlay only the local router mac will be populated in the table.
N7K-1-pod1-S1# show mac address-table Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link VLAN MAC Address Type age Secure NTFY Ports ---------+-----------------+--------+---------+------+----+----------------G 0022.5579.c442 static F F sup-eth1(R) The MAC address in the table is actually the local router MAC, let’s verify this using a physical routed interface already in use (i.e. an uplink interface to the core) N7K-1-pod1-S1# show interface ethernet 1/y mac-address
Interface numbers will change for odd and even POD #s. ---------------------------------------------------------------------------- Check your Network topology to target Interface Mac-Address Burn-in Mac-Address the right interfaces
In OTV we also cache ARP resolution for mac addresses not local to the site and learnt via the overlay. If no traffic was ever sent across the overlay no ARP would have been resolved, and so no entries are cached by the OTV process.
N7K-1-pod1-S1# show otv arp-nd-cache OTV ARP/ND L3->L2 Address Mapping Cache
Let’s now verify the neighbor routers visible via the OTV overlay. We should see two adjacencies here representing the two Nexus 7000 edge devices on the remote sites.
N7K-1-pod1-S1# show otv adjacency Overlay Adjacency database Overlay-Interface Overlay1 : System-ID Dest Addr 0022.5579.1dc2 10.12.11.2 0022.5579.1dc3 10.12.12.2 Adj-State TM_State default default default default Up Time 01:57:41 01:57:46 Adj-State UP UP
The IS-IS protocol used underneath by OTV to advertise MAC addresses over the IP cloud allows to resolve the hostname and gives us a better indication of the neighbor devices:
N7K-1-pod1-S1# show otv isis adjacency OTV-IS-IS process: default VPN: Overlay1 OTV-IS-IS adjacency database: System ID SNPA Level State Site-X 0022.5579.1dc2 1 UP Site-Y 0022.5579.1dc3 1 UP
Hold Time 00:00:25 00:00:23
Interface Overlay1 Overlay1
We can optionally check the OTV overlay information on one of the remote sites. Let’s use the puTTY shortcuts to reach the remote site 7k on Site-X or Site-Y via SSH (see Figure 10).
Figure 10 - SSH puTTY shortcuts to 7K on site-X or site-Y
You will see multiple Overlays and sites depending on how many other students successfully completed this lab in other PODs.
Site-X# show otv isis adjacency OTV-IS-IS process: default VPN: OTV-IS-IS adjacency database: System ID SNPA Site-Y 0022.5579.1dc3 N7K-1-pod1-S1 0022.5579.c442 Overlay1 Level 1 1 State UP UP Hold Time 00:00:27 00:00:07 Interface Overlay1 Overlay1
OTV-IS-IS process: default VPN: Overlay2 OTV-IS-IS adjacency database: System ID SNPA Level State Site-Y 0022.5579.1dc3 1 UP OTV-IS-IS process: default VPN: OTV-IS-IS adjacency database: System ID SNPA Site-Y 0022.5579.1dc3 N7K-4-pod3-S2 0022.5579.be42 OTV-IS-IS process: default VPN: OTV-IS-IS adjacency database: System ID SNPA Site-Y 0022.5579.1dc3 N7K-4-pod4-S2 0022.5579.be43 Overlay3 Level 1 1 State UP UP
Hold Time 00:00:07
Interface Overlay2
Hold Time 00:00:32 00:00:08
Interface Overlay3 Overlay3
Overlay4 Level 1 1 State UP UP Hold Time 00:00:33 00:00:09 Interface Overlay4 Overlay4
OTV-IS-IS process: default VPN: Overlay5 OTV-IS-IS adjacency database: System ID SNPA Level State
OTV-IS-IS process: default VPN: Overlay6 OTV-IS-IS adjacency database: System ID SNPA Level State Site-Y 0022.5579.1dc3 1 UP OTV-IS-IS process: default VPN: OTV-IS-IS adjacency database: System ID SNPA N7K-7-pod7-S1 001b.54c2.b1c2 Site-Y 0022.5579.1dc3 Overlay7 Level 1 1 State UP UP
Hold Time 00:00:06
Interface Overlay6
Hold Time 00:00:33 00:00:07
Interface Overlay7 Overlay7
OTV-IS-IS process: default VPN: Overlay8 OTV-IS-IS adjacency database: System ID SNPA Level State Site-Y 0022.5579.1dc3 1 UP
Hold Time 00:00:08
Interface Overlay8
Because we are only interested to our Overlay, we can also be more specific.
Site-X# show otv isis adjacency <Overlay_id> OTV-IS-IS process: default VPN: Overlay1 OTV-IS-IS adjacency database for Overlay1: System ID SNPA Level State Site-Y 0022.5579.1dc3 1 UP N7K-1-pod1-S1 0022.5579.c442 1 UP
Step 6 Testing Multisite connectivity and mac learning
Let’s now verify connectivity between a Local site (student POD) and the remote sites (Site-X and Site-Y Head-Ends). Every site has a server (actually a VM deployed on an ESX server) which can be used to verify connectivity across the OTV cloud.
Site-X MAC: 0050.5622.2222
Site-X IP: 10.100.0.2
Site-Y MAC: 0050.5633.3333
Site-Y IP: 10.100.0.3
VM where each student operates the Remote Desktop Connection
Figure 11 - POD and Remote Site Address Connectivity
The local VM is the device which hosts the Remote Desktop session that we are currently using to configure the devices. From the current VM let’s open a windows command prompt and from there try to ping the remote IP addresses 10.100.0.2 (Remote Site-X) and 10.100.0.3 (Remote Site-Y) as illustrated at Figure 15 and Figure 14 to verify connectivity toward the remote end sites. Note: the first ping of a new flow maybe lost till learning of the remote MAC happens. Subsequent pings will always be successful.
Last, check on the local Nexus 7000 that addresses of the remote VM servers were learned on the local site and that ARP Table entries, mapping remote IPs and MACs, were cached successfully.
N7K-1-pod1-S1# show mac address-table Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link VLAN MAC Address Type age Secure NTFY Ports ---------+-----------------+--------+---------+------+----+---------------G 0022.5579.d2c2 static F F sup-eth1(R) * 100 0050.5611.1111 dynamic 0 F F Eth2/9 O 100 0050.5622.2222 dynamic 0 F F Overlay1 N7K-1-pod1-S1# show otv arp-nd-cache OTV ARP/ND L3->L2 Address Mapping Cache Overlay Interface Overlay1 VLAN/MAC Address Uptime Layer-3 Address 0100-0050.5622.2222 00:00:35 10.100.0.2
Exp Time Left 00:19:24
Optionally repeat the connectivity test for Remote Site Y
N7K-1-pod1-S1# show mac address-table Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link VLAN MAC Address Type age Secure NTFY Ports ---------+-----------------+--------+---------+------+----+---------------G 0022.5579.d2c2 static F F sup-eth1(R) * 100 0050.5611.1111 dynamic 240 F F Eth2/9 O 100 0050.5622.2222 dynamic 0 F F Overlay1 O 100 0050.5633.3333 dynamic 0 F F Overlay1 N7K-1-pod1-S1# show otv arp-nd-cache OTV ARP/ND L3->L2 Address Mapping Cache Overlay Interface Overlay1 VLAN/MAC Address Uptime Layer-3 Address 0100-0050.5622.2222 00:00:36 10.100.0.2 0100-0050.5633.3333 00:19:30 10.100.0.3
Exp Time Left 00:19:23 00:00:29
You can optionally access the remote servers by simply clicking on the remote Server Site-X and Site-Y puTTY icons (see Figure 12 and Figure 13) to gain console access to both the end stations and perform bidirectional connectivity tests through the Network Overlay. When Logging in into the remote server VMs, use the same login and password used to access you POD and documented in Table 1 - POD Access Details
Congratulations!!! The lab is now complete!
Please LOG OFF from the Windows Machines (Click “Start” on the bottom left corner and “Log Off” right above), do NOT just close the Windows Remote Desktop window.
Recommended Reading
Introductory overview on OTV: http://www.cisco.com/en/US/prod/switches/ps9441/nexus7000_promo.html Cisco Nexus 7000 Series Switches: www.cisco.com/en/US/products/ps9402/index.html Cisco NX-OS Feature Navigator: www.cisco.com/go/nxosnav Cisco NX-OS Home Page: www.cisco.com/go/nxos
Complete Your Online Session Evaluation
Cisco values your input. Give us your feedback! We read and carefully consider your scores and comments, and incorporate them into the content program year after year Go to the Internet stations located throughout the Convention Center to complete your session evaluations Thank you!