Cisco

Cisco ACI bits 'n' bobs

David Chidell

May 30, 2018 • 11 min read

Introduction

I'm fairly new to ACI, and am coming across new things all the time. Here I will provide quick bits and pieces of things I find which are useful to know and have documented.

Content

Erase fabric to factory defaults

When you want to destroy the whole fabric and rebuild it, this is rather handy to know (simple, but useful):

Fabric Switches:

Leaf-1# setup-clean-config.sh
Leaf-1# reload
This command will reload the chassis, Proceed (y/n)? [n]: y

APIC:

DC2-apic3# acidiag touch setup
DC2-apic3# acidiag reboot

APIC Password Recovery

Just in case you forget it, or screw up the APIC. You'll need a USB stick and physical access to the APIC.

Take a blank USB stick and add a single file called "aci-admin-passwd-reset.txt" using your PC.
Insert the USB stick into the APIC and reboot.
When the prompt "Press any key to enter the menu" is displayed, press any key!
Select the version of code the APIC is running.
Push 'e' to edit the boot string
Add "aci-admin-passwd-reset" to the end of the line. Note: No .txt extension!
Press enter to save, and then 'b' to boot the APIC.
The APIC will boot as normal and prompt for a new admin password.

Commands cheat sheet

Clustering User Commands
controller - shows the current cluster size and state of APICs
cd /aci/system/controllers/1/cluster moset administrative-cluster-size (#) moconfig commit - changes the size of the cluster
controller -d -t (ID) - Decommissions the APIC of the given ID
reload [controller|switch] (nodeID) - Reboots the APIC of the given ID
acidiag rvread - shows replica which are not healthy
acidiag rvread (svc) (shard) (replica) - shows the state of one replica
avread - large output which will show cluster size, chassisID, if node is active, and summary of replica health
acidiag fnvread - shows fabric node vector
acidiag avread - shows appliance vector
acidiag verifyapic - verifies APIC hardware
ip link - shows link status
cat /proc/net/bonding/(ID) - shows the status of bond link
acidiag dbgtoken - generates a token which can be decoded by TAC (http://git.insieme.local/passwordGenerator.html) to ssh into the APIC as root using the su - command and typing in the generated password when prompted
show dhcp internal info client - shows dhcp client information to confirm dhcp address from APIC
fabricnode (nodeID) [commission|decommission|wipeout] - commissions, decommissions, or wipes out given node. wipeout will completely wipeout the node including configuration. Use sparingly.

SSL Troubleshooting
openssl s_client -connect (IP):12151 - tries to connect ssl between APIC and Node and gives output of SSL information
zgrep SSL svc_ifc_appliancedirector.bin.log* -shows logging of DME-logs for node
zgrep SSL svc_ifc_policyelem.log* - shows policy-element logs for SSL connectivity
Can also check logs in the /var/log/dme/log directory

Switch Cert Verification
openssl asn1parse /securedata/ssl/server.crt - Next to PRINTABLESTRING, it will list Insieme or Cisco Manufacturing CA. Cisco means new secure certs are installed, Insieme means old unsecure are installed
openssl x509 -noout -issuer -subject -dates -in /securedata/ssl/server.crt - Shows start and end dates of certificate. Must be within range for APIC to accept
act_util key_pair show (#) - Shows keypairs of specified cert
openssl x509 -in cert.crt -text –no out - Shows the certificate that is in use. Must be root.

Switch Diagnostics
show module internal event-history module (#) - shows bootup tests and diagnostics of given module
show diagnostic content module (ID) - shows ongoing tests of given module
show diagnostic result module [all|(moduleID)] - shows diagnostic result of given module or all modules
show diagnostic result module (moduleID) test (testID) detail - shows diagnostic result of given test on given module
show diagnostic internal [diagmgr|diagclient|port_lb] - show debug informatino for the diagnostic modules

Debug Commands
debug platform internal emon [heartbeat|kfsm|stats|traffic] - shows debug output of given argument
debug platform internal emon [heartbeat|kfsm|stats|traffic] [enable|disable] - enables/disables given argument on all modules
debug platform internal emon [heartbeat|kfsm|stats|traffic] interval get - gets the interval of given argument
debug platform internal emon stats get (ID) - EPC mon statistics
debug platform internal emon kfsm state get (ID) - EPC mon statistics
debug platform internal marvell switch [0|1] status - EOBC/EPC switch status (0: EOBC, 1: EPC)
debug platform internal broadcom switch status - SC card broadcom switch status

Insieme ELTM VRF, VLAN, Interface Commands
debug system internal [eltm|eltmc] trace output file - dumps ELTM trace to output file
show system internal [eltm|eltmc] info trace - dumps eltm trace to console
show system internal [eltm|eltmc] info vrf (vrf) - shows vrf table of given vrf
show platform internal ns forwarding segments -
show platform internal ns forwarding epgs -
cat summary - vrf summary, shows ID, pcTag, scope
show system internal eltmc info vlan brief - shows vlan information. Can substitute (brief) for a vlan ID
show sytem internal eltmc info interface (interface ID) -

OSPF CLI Commands
show ip ospf neighbors vrf (vrf|all) - shows OSPF neighbors of given vrf
show ip ospf route vrf (vrf|all) - shows OSPF routes of given vrf
show ip ospf interface vrf (vrf|all) - shows ospf interfaces of given vrf
show ip ospf vrf (vrf|all) - shows ospf information of given vrf
show ip ospf traffic vrf (vrf|all) - shows ospf traffic of given vrf

External Connectivity
show ip arp vrf (vrf) - shows arp entries for given vrf
show ip ospf neighbors vrf (vrf) - shows ospf neighbors for given vrf
show bgp sessions vrf (vrf) - shows bgp sessions/peers for given vrf
show ip ospf route vrf (vrf) - shows ospf routes for given vrf
show bgp ipv4 unicast vrf (vrf) - shows bgp unicast routes for given vrf
show ip static-route vrf (vrf) - shows static routes for given vrf
show ip route vrf (vrf) - shows routes for given vrf
l3 defip show - shows external LPMs
l3 egress show - shows next hops towards NorthStar ASIC or external router
show platform internal ns table mth_lux_slvd_DHS_HigigDstMapTable_memif_data ingress - HigigDstMapTable Indexed using DMOD/DPORT coming from T2. Provides a pointer to DstEncapTable.
show platform internal ns table mth_lux_slvg_DHS_DstEncapTable_memif_data ingress - DstEncapTable Indexed using the HigigDstMapTable’s result. Gives tunnel forwarding data.
show platform internal ns table mth_rwx_slva_DHS_RwEncapTable_memif_data ingress - RwEncapTable Indexed using the HigigDstMapTable’s result. Gives tunnel encap data.

ISIS Fabric Unicast Debugging
show isis protocol - shows ISIS statistics
show isis adjacency [detail] vrf (vrf) - shows ISIS adjacencies for given vrf. Can also add detail
show lldp neighbor - shows lldp neigbor status
show interface (interface ID) - shows interface status information and statistics
show isis database [detail] vrf (vrf) - shows isis database, can also add detail
show isis route vrf (vrf) - shows isis route information
show isis traffic vrf (vrf) - shows isis traffic information
show isis dtep vrf (vrf) - shows all discovered tunnel end points
show isis statistics (vrf) - shows isis statistics of given vrf
show isis event-history [detail] - shows isis event history
show isis internal mem-stats [detail] - shows isis memory statistics
show tech-support-service isis - provides isis tech-support output for TAC

ASIC Platform Commands
show platform internal [ns|alp] mac asic [0|1] - shows the MAC port status
show platform internal [ns|alp] counters mac asic [0|1] - shows the MAC port counters
show platform internal [ns|alp] counters asic-block [all|bax|lbx|lux|prx|qsx|rwx|scx|top] - shows ASIC block counters for given ASIC. Can also add [detail] for more details
show platform internal [ns|alp] interrupts - shows interrupts for given ASIC

ASIC Platform Commands - T2 Specific
show c rpkt - shows receive counters for T2
show c tpkt - shows transmit counters for T2
show c xe12 - shows per port packet type counters
g chg ing_event_debug - shows ingress drop counters
g chg_egr_drop_vector - shows egress drop counters
cstat xe17 - checking the stats for above command

ASIC Platform Commands - NS Specific
show platform internal counters port - shows port counters
show platform internal counters port internal - shows internal port counters
show platform internal counters vlan - shows vlan counters
show platform internal counters tep - shows per-tunnel counters
show platform internal ns counters asic-block all - shows ASIC block counters
show platform internal ns forwarding list - shows well-defined tables

Fabric Multicast - General
show isis internal mcast routes ftag - shows currecnt state of FTAG, cost, root port, OIF list
show isis database mgroup detail vrf (vrf) - shows GM-LSP database
show isis internal mcast routes gipo - shows GIPO routes, Local/transit, OIF list
show isis internal mcast statistics - shows topology and compute stats, MRIB update stats, Sync+Ack packet stats, Object store stats
show isis event-history mcast - shows isis multicast event history logs
show isis event-history mcast-convergence - more detailed than above command, specifically dealing with forwarding events and forwarding updates

Fabric Multicast Debugging - MFDM
show forwarding distribution l2 multicast - flood/OMF/GIPi membership
show forwarding distribution l2 multicast vlan (vlanID) per BD
show forwarding distribution l2 multicast gipi - GIPi membership
show forwarding distribution l2 multicast gipi (IP) - specific
show forwarding distribution l2 multicast gipi vlan (vlanID) - per BD
show forwarding distribution l2 multicast gipi (IP) vlan (vlanID) - specific per BD
show forwarding distribution l2 multicast flood - flood membership
show forwarding distribution l2 multicast flood vlan (vlan ID) - per BD
show forwarding distribution l2 multicast omf - OMF membership
show forwarding distribution l2 multicast omf vlan (vlan ID) - per BD
show system internal forwarding distribution multicast ipmc - IPMC membership
show system internal forwarding distribution multicast ipmc 0x3 - specific IPMC
show forwarding distribution multicast ipmc-sw
show forwarding distribution multicast ipmc-sw (ID)

Fabric Multicast Debugging - L2 Multicast
show system internal forwarding l2 multicast - flood/OMF/GIPi membership
show system internal forwarding l2 multicast vlan (vlanID) - per BD
show system internal forwarding l2 multicast gipi - GIPi membership
show system internal forwarding l2 multicast gipi (IP) - specific
show system internal forwarding l2 multicast gipi vlan (vlanID) - per BD
show system internal forwarding l2 multicast gipi (IP) vlan (vlanID) - specific per BD
show system internal forwarding l2 multicast flood - flood membership
show system internal forwarding l2 multicast flood bd (bdID) - per BD
show system internal forwarding l2 multicast met - MET membership
show system internal forwarding l2 multicast met (ID) - specific MET
show system internal forwarding l2 multicast met flood - flood MET
show system internal forwarding l2 multicast met gipi - GIPi MET
show system internal forwarding l2 multicast met gipi bd (bdID) - per BD
show system internal forwarding l2 multicast met gipi (IP) bd (bdID) - specific per BD
show system internal forwarding l2 multicast ipmc - IPMC membership
show system internal forwarding l2 multicast ipmc (ID) - specific IPMC

Fabric Multicast Debugging - MRIB
show ip mroute vrf (vrf) - shows IP multicast routing table for given vrf

Fabric Multicast Debugging - MFIB
show ip fib mroute ftag - shows FTAGs
show forwarding vrf all multicast route - shows GIPo routes

Fabric Multicast Debugging - IGMP
show ip igmp snooping groups - shows multicast route information in IGMP
show ip igmp snooping mrouter - shows multicast router information IGMP
show ip igmp snooping encap-db - FD to BD vlan mapping. IGMP gets FD and G from Istack. It needs to know the BD to create (BD, G)
show ip igmp snooping vlan (vlanID) - verify BD membership of a port in IGMP. Only when ports are part of BD joins are processed
show ip igmp snooping vtep-if-db - verify the tunnel to IF mapping in IGMP. IGMP uses this to get the groups on VPC and only sync them.

Fabric Multicast Debugging - MFDM
show forwarding distribution ip multicast route vrf (vrf) - shows IPv4 multicast routing table for given vrf
show forwarding distribution multicast vlan_db - Verify FD to BD vlan mapping. MFDM gets (FD,port) memberships from vlan_mgr and uses this information go create BD floodlists.
show forwarding distribution multicast bd_gipo - BD to GIPO mapping. GIPO is used by Mcast in Fabric
show forwarding distribution multicast epg_gipo_prime - FD-vxlan to GIPO mapping
show forwarding distribution multicast vtep_if_db - tunnel to phy mapping

Fabric Multicast Debugging - M2rib
show l2 mroute - shows multicast route information in M2rib
show l2 mroute omf - shows multicast route informatino in M2rib

Fabric Multicast Debugging - PIXM
show system internal pixm info ltl-range start-ltl 0x0 ltl-cnt 4000 - RID to IPMC mapping. IFIDX is RID and LTL is IPMC

Fabric Multicast Debugging - VNTAG Mgr
show system internal vntag dvif-allocation - IPMC to DVIF mapping. LTL is IPMC

EP Announce - Debugging
show system internal epm announce
show system internal epm counters announce
show system internal epm vlan (vlanID) detail
show system internal epm vrf (vrf) detail
show system internal epm periodic
show system internal epm endpoint all

iBash CLI
show mac address-table
show endpoint [summary|address|interface|vlan|vrf] - show endpoint information

BCM Table Dump
bcm-shell-hw "l2 show"
bcm-shell-hw "l3 l3table show"

Fabric QoS Debugging - CoPP CLI
show copp policy
show system inernal aclqos brcm coppp entries unit 0 - CoPP statistics (red = dropped, green = allowed)
show system internal qos classes - shows QoS classes configured
show system internal qos vlan all - shows QoS classes/policices configured per vlan
show system internal qos ppf [pinst|nodes] - shows ppf details
show system internal aclqos qos classes - shows QoS classes configured in hardware
show system internal aclqos qos vlan (vlanID) - shows the QoS DSCP/dot1p policy configured for a vlan in HW
show system internal aclqos qos policy summary - shows QoS DSCP/dot1p policy summary
show system internal aclqos qos policy detail - shows QoS DSCP/dot1p policy in detail
show system internal aclqos brcm tcam entries unit 0 group [efp-bpdu|efp-ctrl-pol|efp-mark|ifp-ctrl|ifp-dscp|ifp-elmc-vleaf|ifp-span-port-vlan|ifp-span-port-vlan-egress|ifp-span-vlan-egress|ifp-vni-udf|vfp-vni] - shows T2 TCAM entries for specified group
show platform internal counters port (#) - shows QoS counters on each port
show platform internal counters port internal (#) - shows QoS counters on each port (internal)
show platform internal counters class (#) - shows QoS counters for each class for all ports

MCP CLI
show mcp internal info global - shows the edge port config on the HIF (FEX) ports, the internal VLAN mapping and the STP TCN packet statistics received on the fabric ports
show mcp internal info interface [all|interfaceID] - shows mcp information by interface
show mcp internal info stats interface - shows stats for all interfaces
show mcp internal info vlan [all|vlanID] - shows mcp information per vlan
show mcp internal stats vlan - shows stats for all vlans
show mcp internal info msti [all|(region name) (instance ID)] - shows mcp information per msti region
show mcp internal info stats msti - shows stats for all msti regions

iTraceroute CLI
itraceroute (destinationIP) (pld-size) - node traceroute
itraceroute (destinationIP) vrf (vrf) encap vlan (vlan-encap) payload (pld-size) - Tenant traceroute for vlan encapped source EP
itraceroute (destinationIP) vrf (vrf) encap vxlan (vxlan-encap) dst-mac (dst-mac) payload (pld-size) - Tenant traceroute for vxlan encapped source EP

ELAM Setup and debugging (follow commands in order)
debug platform internal ns elam asic (#) - starts ELAM on given ASIC
trigger init ingress in-select 3 out-select 0 - sets trigger for ELAM
set outer l2 dst_mac (destination mac) src_mac (source mac) - sets source and destination mac addresses
start - Starts capture
status - shows capture status
report - shows report of the capture

VMM Troubleshooting
show vmware controllers - shows VM controllers and their attributes such as IP/hostname, state, model, serial number
show vmware domain mininet (name) inventory - shows hypervisor inventory of given VM controller
show vmware domain mininet (name) [inventory|policy|status]
show vmware domain mininet (name) inventory [hypervisors|portgroups|virtual-machines|virtual-switches]

TOR Sync Troubleshooting
netstat -tp | grep epm
tcpdump -i kpm_inb
show system internal epm vpc
show system internal epm counters vpc
show system internal epm counter zmq
show system internal epm announce
show system internal epm counters announce
show system internal epm vlan (vlanID) [detail] - can see which VLAN is learn disable
show system internal epm vrf (vrf) [detail] - can see which VLAN is learn disable
show system internal epm periodic - see if timer is attached on the VLAN/vrf
show system internal epm counters all
show system internal epmc counters all

OpFlex Debugging
vemcmd show openflex - shows if OpFlex is online (status = 12 means OpFlex is online, remoteIP is anycast IP, intra vlan is vlan used by VTEP, FTEP IP is the iLeaf's IP)
vem status - check if DPA is running
vemcmd show sod
vemcmd show port - uplinks and vtep should be in forwarding state. PC-LTL of uplink port should be non-zero
vemcmd show pc - Check port channel type
vemcmd show lacp - if port channel type is LACP, can use this command to see the individual uplink LACP state
esxcfg-vmknic -l - verify if the VTEP received a valid DHCP IP address

SPAN Debugging
vemcmd show span

BPDU Debugging
vemcmd show card - shows if BPDU Guard/Filter is enabled or disabled
vemcmd show bpdu-stats - check if the bpdu-drop stats are incrementing on the uplinks/virtual ports

VEM Misc Commands
vemcmd show openflex - show channel status
vemcmd show port - check port status
vemcmd show bd - check per EPG flood lists
vemcmd show epp multicast - check vLeaf multicast membership
vemcmd show stats - show packet stats
vemcmd show packets - show packet counters
vemcmd show port-drops - show packet drops by ports
vemlog debug sfport all - debug vxlan packet path
vemlog debug sflayer2 all - debug vxlan packet path
vemlog show all - show above logging output
vempkt capture [egress|pre-ingress]
vempkt clear
vempkt start
vempkt stop
vempkt display brief all
vempkt display detail entry (#)
vempkt cancel capture all

FEX Troubleshooting
show fex - shows all FEXs and their states
show fex (#) [detail] - gives detailed stats of given FEX
show environment fex - gives environmental stats of FEX
show fex transceiver
show fex version - shows FEX version
show interface fex-fabric - shows FEX fabric interface information
show logging level fex - shows logging information for FEX
show interface transceiver fex-fabric - shows transceiver information for FEX
show system reset-reason fex - show FEX reset reason
show module fex - shows FEX module information
show system internal fex log | grep (anything) - shows debugging information and you can grep to find what you want
show system internal fex internal event-history msgs - use to find out which service is failing the sequence and you can debug that process further.

Introduction

Content

Erase fabric to factory defaults

APIC Password Recovery

Commands cheat sheet

Sign up for more like this.