Multichassis link aggregation (MLAG) lets IT teams multihome servers to two or more physical switches in an MLAG group while still allowing all links to forward traffic bi-directionally. MLAG typically uses 802.3ad (LACP) as the control protocol. As there is no single standard for MLAG deployment, network vendors offer it to their customers in various ways. One commonly deployed MLAG technology from Cisco is virtual port-channel (vPC), available in its Nexus line of data center switches. For more on MLAG and other network architecture options, download the report The Virtual Network.
In the Nexus world, the most common use cases for vPC are to create high-bandwidth interswitch links spanning multiple physical switches, and to uplink virtualization platforms such as blade servers and VMware hosts. 10-Gbps links are expensive, and MLAG helps ensure you can use them all. Network engineers who want to deploy vPC should study Cisco's thorough, scenario-driven vPC design guides, one of which can be found here. While the process of building a vPC domain and creating vPCs does not generate especially complex configuration code, the design implications of vPC should be well understood before using it in a production environment.
The rest of this article demonstrates the process of creating a vPC domain between two Cisco Nexus 5500 switches running NX-OS 5.2. Creating the vPC domain is the necessary foundation before creating host-facing vPCs. Note that Cisco supports only two switches in a vPC domain as of this writing. Other MLAG implementations, especially those based on stacking technologies, can support more than two switches in an MLAG domain.
Key points about this configuration:
-The MGMT interface on each Nexus switch is used for the vPC peer-keepalive (heartbeat) link. This is because the peer-keepalive does not have to be a highly redundant link. Cisco recommends using the MGMT interface for this function, which does not prevent you from using it for other functions such as SNMP and TACACS. If the MGMT interfaces lose contact with one other, that doesn't automatically affect the status of the vPC domain. The peer-link must also have problems before vPC domain members will assume the topology is breaking down and begin to take action to preserve the forwarding path.
-Port-channel10 is used as the vPC peer-link between the Nexus switches in this example.
-The vpc domain number will always be the same for the two switches in the domain, or the vPC domain will not form. If switches in one vPC domain are to be plumbed to switches in a different vPC domain, then the vPC domain number you assign each pair should differ. For example, if I assign the number "10" to two switches in a vPC domain, I would assign a different number, such as "11," to the two adjacent switches in a second vPC domain.
-The system-priority command is related to LACP negotiation of active vs. standby links in LACP bundles of more than eight physical links. While it is not required that you configure this value, Cisco recommends you set it so that the Nexus vPC switches can decide which links are to be active vs. standby. Lower system-priorities take precedence. This value must match on both switches, or else the vPC domain will not form.
-The role priority command sets a vPC switch priority. Lower role-priorities take precedence. This helps determine which switch will shut down its vPC ports to prevent a topology loop if vPC communications are interrupted.
-The peer-link interface does not require large amounts of bandwidth, as vPC traffic management and port-channel hashing keeps dual-attached end points talking via one of the two vPC domain switches. The vPC peer link will only be used in the case of an unbalanced uplink (a device is plumbed to two switches, but one link has failed), or possibly in the case of single-attached hosts. Cisco strongly advocates that single attached hosts are connected to a vPC domain via a dual-homed switch or fabric extender.
-The peer-gateway command allows a vPC peer to route a packet that, due to HSRP priorities, should have been forwarded to the vPC peer switch for routing services instead of being routed locally. This overcomes hosts that don't use the virtual MAC for an HSRP IP, but instead uses the physical MAC. Jeremy Filliben has an excellent writeup on the subject.
-The auto-recovery reload-delay command allows a vPC switch that reloads, but finds its peer gone once it comes back up, to move to an active state after the delay time has expired. Without this command, vPC ports would remain shut down. Also, this command allows the secondary vPC switch to bring its ports back to active, if after a peer-link failure, the primary switch also fails.
-Finally, note that not every possible command or design consideration is mentioned in this article. I'm providing the reader with an introduction to vPC design consideration and domain configuration. Think of this is a good starting point, but then take your own specific data center requirements and apply vPC to them.
The Code
----------------------------------------------
NEXUS-SWITCH1 | MGMT0=10.10.10.1
----------------------------------------------
conf t
!
! NX-OS "features" are used to enable functionality
! that is disabled by default.
feature lacp
!
! The port-channel load-balance command determines the
! hashing algorithm used to determine what physical
! path a flow will be sent down.
port-channel load-balance ethernet source-dest-port
!
! To maximize redundancy and minimize single points of
! failure, I recommend spreading physical links in a
! port-channel across multiple internal ASICs. In the
! case of the Nexus 5596 used in this example,
! contiguous groups of 8 ports are mapped to a single
! ASIC – 1-8, 9-16, etc.
interface e1/1,e1/9
switchport
channel-group 10 mode active
exit
!
interface Port-Channel10
des UPLINK NEXUS-SWITCH2 | 10.10.10.2 | Po10
switchport mode trunk
no shut
exit
!
! With CDP and LLDP, detailed port descriptions might
! seem superfluous, but I find them useful as references
! when monitoring with an NMS.
interface e1/1
des UPLINK NEXUS-SWITCH2 | 10.10.10.2 | Eth1/1
exit
!
interface e1/9
des UPLINK NEXUS-SWITCH2 | 10.10.10.2 | Eth1/9
exit
!
! This enables the vPC feature set. The balance of the
! commands below are discussed in the opening paragraphs.
feature vpc
!
vpc domain 10
peer-keepalive destination 10.10.10.2
system-priority 8192
role priority 8192
peer-gateway
auto-recovery reload-delay 240
exit
!
interface Port-channel10
vpc peer-link
vpc 10
exit
!
exit
!
copy run start
----------------------------------------------
NEXUS-SWITCH2 | MGMT0=10.10.10.2
----------------------------------------------
conf t
!
feature lacp
!
port-channel load-balance ethernet source-dest-port
!
interface e1/1,e1/9
switchport
channel-group 10 mode active
exit
!
interface Port-Channel10
switchport mode trunk
no shut
exit
!
interface e1/1
des UPLINK NEXUS-SWITCH1 | 10.10.10.1 | Eth1/1
exit
!
interface e1/9
des UPLINK NEXUS-SWITCH1 | 10.10.10.1 | Eth1/9
exit
!
feature vpc
!
vpc domain 10
peer-keepalive destination 10.10.10.1
system-priority 8192
role priority 16384
peer-gateway
auto-recovery reload-delay 240
exit
!
interface Port-channel10
vpc peer-link
vpc 10
exit
!
exit
!
copy run start
Verification
Network engineers should verify completed tasks. To verify that a vPC domain has formed successfully, review the output of the "show vpc" command. While the output below shows a functional vPC domain, error messages would indicate issues to be investigated.
NEXUS-SWITCH1# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 10
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status: success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 0
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 Po10 up 1
NEXUS-SWITCH1#
Ethan Banks is CCIE #20655, a host of the Packet Pushers podcast, and IT practitioner since 1995. He can be reached at @ecbanks.
