Monitoring CSM Services and finding MIB

CSM Quick CMDS

show module contentSwitchingModule 3 stats
show module contentSwitchingModule 3 variable

FIND THIS MIB

SR_C6500#sh module contentSwitchingModule all vservers  | inc 10.111.111.143
MYPLTCAS_VIP110 SLB   TCP  10.111.111.143/32:110    ALL  OPERATIONAL  95
MYPLTCAS_VIP143 SLB   TCP  10.111.111.143/32:143    ALL  OPERATIONAL  157
MYPLTCAS_VIP443 SLB   TCP  10.111.111.143/32:443    ALL  OPERATIONAL  9806
MYPLTCAS_VIP80  SLB   TCP  10.111.111.143/32:80     ALL  OPERATIONAL  0
MYPLTCAS_VIP993 SLB   TCP  10.111.111.143/32:993    ALL  OPERATIONAL  179
MYPLTCAS_VIP995 SLB   TCP  10.111.111.143/32:995    ALL  OPERATIONAL  52
SR_C6500#


popnnm06#  snmpwalk -v 1 -c public 10.111.2.10 1.3.6.1.4.1.9.9.254 | grep
popnnm06#  snmpwalk -v 1 -c public 10.111.2.10 1.3.6.1.4.1.9.9.254 | grep 10.111.111.143
popnnm06#  snmpwalk -v 1 -c public 10.111.2.10 1.3.6.1.4.1.9.9.161.1.4 | grep 10.111.111.143
popnnm06#  snmpwalk -v 1 -c public 10.111.2.10 SNMPv2-SMI::enterprises.9.9.161.1.3.1.1.16.3 | grep "Counter32: 8[0-9][0-9][0-9]"

# REALS
snmpwalk -v 1 -c public 10.111.2.10 enterprises.9.9.161.1.3.1.1.5.3
snmpwalk -v 1 -c public 10.111.2.10 enterprises.9.9.161.1.3.1.1.7.3 | grep 10.111.111
snmpwalk -v 1 -c public 10.111.2.10 enterprises.9.9.161.1.3.1 | grep "Counter32: [76][0-9][0-9][0-9]"
snmpwalk -v 1 -c public 10.111.2.10 .1.3.6.1.4.1.9.9.161.1.3.1.1.5 | grep 10.111.111
snmpwalk -v 1 -c public 10.111.2.10 SNMPv2-SMI::enterprises.9.9.161.1.3.1.1.5.3.12.80.84.65.83.80.67.65.83.45.49.49.48

# VSERVERS
snmpwalk -v 1 -c public 10.111.2.10 1.3.6.1.4.1.9.9.161.1.4.1.1.17.3

FIND VSERVER MIB
SR_C6500#sh module contentSwitchingModule all vservers  | inc 10.111.111.172
SR_C6500#sh module contentSwitchingModule all vservers name MYSERVERASHUB-110 config
Depois de saber os reals, procurar o ip do real na config. Por exemplo neste caso seleccionei:
  PTAEXCASHUB01-10.111.111.166
# find real que comece por 10.111.111.166 com a porta 110
snmpwalk -v 1 -c public 10.111.2.10 .1.3.6.1.4.1.9.9.161.1.3.1.1.5 | grep 10.111.111.166 | grep "110 = G"
SNMPv2-SMI::enterprises.9.9.161.1.3.1.1.5.3.13.80.84.65.67.65.83.72.85.66.45.49.49.48.10.111.111.166.110 = Gauge32: 18
                                               80.84.65.67.65.83.72 | grep 49.48.
# find vserver:
snmpwalk -v 1 -c public 10.111.2.10 1.3.6.1.4.1.9.9.161.1.4.1.1.17.3 | grep 80.84.65.67.65.83.72 | grep 49.48
SNMPv2-SMI::enterprises.9.9.161.1.4.1.1.17.3.13.80.84.65.67.65.83.72.85.66.45.49.49.48 = Gauge32: 105


# find real que comece por 10.111.111.160 com a porta 143
snmpwalk -v 1 -c public 10.111.2.10 .1.3.6.1.4.1.9.9.161.1.3.1.1.5 | grep 10.111.111.160 | grep "143 = G"
SNMPv2-SMI::enterprises.9.9.161.1.3.1.1.5.3.12.80.84.65.83.80.67.65.83.45.49.52.51.10.111.111.160.143 = Gauge32: 14
                                               80.84.65.83.80.67.65.83 | grep 49.52.51
# find vserver:
snmpwalk -v 1 -c public 10.111.2.10 1.3.6.1.4.1.9.9.161.1.4.1.1.17.3 | grep 80.84.65.83.80.67.65.83 | grep 49.52.51
result:
SNMPv2-SMI::enterprises.9.9.161.1.4.1.1.17.3.15.80.84.65.83.80.67.65.83.95.86.73.80.49.52.51 = Gauge32: 101
1.3.6.1.4.1.9.9.161.1.4.1.1.17.3.15.80.84.65.83.80.67.65.83.95.86.73.80.49.52.51 -> MIB VSERVER

Datacenter Network Capacity Management

Here is a complex topic: Capacity Management inside the Datacenter.

There is daily operation and monitoring, and there is capacity management. They will touch themselves in some points but they have completly diferent perspectives:
I have been doing capacity KPIs for the past 8 years. To be honest, I believe that now it is the first time I am doing it right. Here are some diferences:

Operation:
- Realtime monitoring
- Alarm driven
- Focused in detail
- We are worried with uptime

Capacity Management:
- Its a recurrent process
- It is weekly based
- We are worried with provision capacity
- We are worried with long term stability
- We want to have a big picture of all infrastruture.

So, the first questions that normally arise are:
- How can I extract my data
- What are my limits
- Which tools can I use.

Well , the correct questions should be "How can I setup a process that will take actions on important findings triggered by capacity management". If you can answer this question and execute in a process then , you already have capacity management in place. Now, all you need is data and analytics to feed this process. Believe me that setting the process is the hard part because it is normally taken as a less priority task that can always be done later.

So, after the process is setup and running, which means, you will have a weekly review, meeting minutes weekly report that will show you are doing it will all  updated findings, than you will have something for an auditor in a ISO 20000 certification.

But, what you are probably trying to get is, what should I measure and how. Getting the data and finding the limits is also not easy.
If you work in a service provider, you will have dozens of networking device vendores, diferent models, diferent version in the same models (templates), and all will have diferent limits and measurements. Also, the configuration applied to simliar appliances will have diferent measurements because the configuration is diferent. For example, a Cisco ASA configured with multi context must have diferent measurements then a non virtualized chassis.








(will continue ...)

ssh login delays 12 seconds

Hi,

Been hitting my head off for the past hour or so to resolve my ssh login lag of about 12 seconds to a centos server.
The fix can be done in the client if you are using putty by disabling GSAPI option:
   Connection/ssh/auth/GSAPI
   Attemp t GSAPI authentication (SSH-2 only)

To fix it permanently, then I believe that you will have to do it in the server. Let's try ...
I am using centos:

Did the following in /etc/ssh/sshd_config:
# GSSAPI options
GSSAPIAuthentication no
#GSSAPIAuthentication yes

It worked! Now, I have no 12 second delay!

Start service at boot in linux

chkconfig -> check the services status at boot
chkconfig snmpd on -> Make snmpd start at boot

Nexus 9k Scripting - Part I

Today I have spent some minutes trying out new ways to automate Nexus 9k.

Out of the box, you can get in to Nexus 9k linux OS. Jut do the following:

Switch(config)# feature bash-shell

Switch# run bash
bash-4.2$
bash-4.2$
bash-4.2$
bash-4.2$ cat /proc/cpu
cat: /proc/cpu: No such file or directory
bash-4.2$ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 58
model name      : Intel(R) Core(TM) i3-3227U CPU @ 2.50GHz
stepping        : 9
microcode       : 0x19
cpu MHz         : 2500.000
cache size      : 4096 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips        : 5001.98
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 58
model name      : Intel(R) Core(TM) i3-3227U CPU @ 2.50GHz
stepping        : 9
microcode       : 0x19
cpu MHz         : 2500.000
cache size      : 4096 KB
physical id     : 0
siblings        : 4
core id         : 1
cpu cores       : 2
apicid          : 2
initial apicid  : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips        : 5001.98
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 6
model           : 58
model name      : Intel(R) Core(TM) i3-3227U CPU @ 2.50GHz
stepping        : 9
microcode       : 0x19
cpu MHz         : 2500.000
cache size      : 4096 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 2
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips        : 5001.98
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 58
model name      : Intel(R) Core(TM) i3-3227U CPU @ 2.50GHz
stepping        : 9
microcode       : 0x19
cpu MHz         : 2500.000
cache size      : 4096 KB
physical id     : 0
siblings        : 4
core id         : 1
cpu cores       : 2
apicid          : 3
initial apicid  : 3
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips        : 5001.98
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

bash-4.2$

! If you want to be root just do the following
bash-4.2# sudo su root
bash-4.2# whoami
root
bash-4.2#
bash-4.2# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       1.0G  485M  540M  48% /
none            1.0G  468M  557M  46% /isan
none             40M   40M     0 100% /nxos/tmp
none             50M  8.1M   42M  17% /var/log
none            5.0M   12K  5.0M   1% /var/home
none            300M  156K  300M   1% /var/tmp
none            3.5G  2.0M  3.5G   1% /var/sysmgr
none            600M  201M  400M  34% /var/sysmgr/ftp
none            100M     0  100M   0% /var/sysmgr/srv_logs
none             10M     0   10M   0% /var/sysmgr/ftp/debug_logs
none            3.0G  310M  2.7G  11% /dev/shm
none            600M     0  600M   0% /volatile
none            5.0M   12K  5.0M   1% /debug
/dev/sda5       4.7G  139M  4.3G   4% /mnt/cfg/0
/dev/sda6       4.7G  139M  4.3G   4% /mnt/cfg/1
/dev/sda4        21G  474M   20G   3% /bootflash
none            400M  3.3M  397M   1% /var/sysmgr/startup-cfg
/dev/sda3       4.7G  139M  4.3G   4% /mnt/pss
/dev/sda2       2.4G   69M  2.2G   4% /mnt/plog
/dev/sda7        21G  254M   20G   2% /logflash
bash-4.2#

bash-4.2$ sudo su -
Linux#
Linux#
Linux#
Linux#
Linux#
Linux#
Linux# whoami
root
Linux#
Linux#
Linux#
Linux#
Linux#
Linux#
Linux#
Linux# cat /proc/meminfo
MemTotal:       16402672 kB
MemFree:        12235656 kB
Buffers:          471856 kB
Cached:          1947636 kB
SwapCached:            0 kB
Active:          2382132 kB
Inactive:        1276432 kB
Active(anon):    1552648 kB
Inactive(anon):  1240200 kB
Active(file):     829484 kB
Inactive(file):    36232 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 4 kB
Writeback:             0 kB
AnonPages:       1239144 kB
Mapped:           174804 kB
Shmem:           1553800 kB
Slab:              83676 kB
SReclaimable:      34568 kB
SUnreclaim:        49108 kB
KernelStack:        3440 kB
PageTables:        16296 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     8201336 kB
Committed_AS:   10795204 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      887816 kB
VmallocChunk:   34358789872 kB
HardwareCorrupted:     0 kB
DirectMap4k:       14336 kB
DirectMap2M:    16754688 kB
Linux# run bash ps -el
-su: run: command not found
Linux# python


bash-4.2$ pwd
/bootflash/scripts
bash-4.2$ vi myteste-tcl
! write your script
bash-4.2$ cat myteste-tcl
puts "my first tcl script in Nexus 9k"
bash-4.2$ exit
Switch#  tcl bootflash:/scripts/myteste-tcl
my first tcl script in Nexus 9k
Switch#

Cisco 7600 with Total output drops on an interface

interface G1/1
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 6469513


Diagnose:
show counters interface GigabitEthernet 1/1  | inc Dis
 8.                       ifInDiscards = 0
10.                      ifOutDiscards = 116890291
11.            txDelayExceededDiscards = 0
23.                         InDiscards = 0
24.                        OutDiscards = 116890273
48.              DelayExceededDiscards = 0


https://fasterdata.es.net/network-tuning/router-tuning/cisco/


interface G1/1
  hold-queue 4096 out
  hold-queue 1024 in

IOS-XR ASR 9k dhcp relay

When trying to configure dhcp in ASR 9k, got into some problems.
First, ASR 9k does not have dhcp server, at least till version 4.3.2. You will always need a external dhcp server. 
So, when configuring dhcp relay to my corporate dhcp, here's what I found:

Does not work:
interface  bvi2
   ipv4 helper-address vrf CORP 10.1.100.10
   ipv4 helper-address vrf CORP 10.2.100.10

Works:
dhcp ipv4
 profile CORP_DHCP relay
  helper-address vrf CORP 10.1.100.10
  helper-address vrf CORP 10.2.100.10
!
dhcp ipv4 interface bvi2 relay profile CORP_DHCP


Hope it helps! Regards

Monitor BGP Peers via SNMP in Nexus 7k 7000

ASAv Initial Configuration

Here is an inital configuration for ASAv in order to allow ssh access and start working with it:

INIT CONFIG ASAv MANUAL PROVISION

What do you need:
hostname: MYASAVxxx
mgmt ip: 10.1.2.x
interface: mgmt0/0 interface must be in mgmt vlan


hostname MYASAVxxx
interface Management0/0
 management-only
 nameif management
 security-level 100
 ip address 10.1.2.xxx 255.255.255.0
!
route management 0.0.0.0 0.0.0.0 10.1.2.1 1
ssh 0.0.0.0 0.0.0.0 management
username restclient password restclient encrypted privilege 15
aaa authentication enable console LOCAL
aaa authentication ssh console LOCAL
username restclient password restclient privilege 15 
crypto key generate rsa modulus 1024

Cisco ASR IOS-XR - How to remove comments

IOS-XR accepts all ! comands and they stay in the configuration !

If execute the command
! Hi all
l2vpn


The comments stays there and it is associated with the next command you execute.
Now, how can we remove it!

Just do it with the "clear comment" command.
Example:

router static
 vrf MYVRF
  address-family ipv4 unicast
   ! This is server A route
   10.1.1.1/32 2.2.2.2
commit


router static
 vrf MYVRF
  address-family ipv4 unicast
   clear comment
   10.1.1.1/32 2.2.2.2
commit

Apache forbidden message for files with underscore

In my automation server for my cisco equipament, after reboot, things stopped working with apache. After some time troubleshooting I found out that images with underscore in the file name would be forbidden while images wihout would be shown by apache. Really strange!

After trying everything:
- permissions
- directory http files
- etc

The problem was with SELINUX.
Executing echo 0 > /selinux/enforce

To be permanent, do the folling:
On /etc/sysconfig/selinux
change from:
SELINUX=enforcing

to :
SELINUX=disabled

Then, reboot again!
Good luck

Cisco opens up EIGRP

Wow! This is a strange decision from Cisco. So many years closing this protocol to other vendors and now sundently they open it to IETF. I guess that the conclusion is: Now a days, networking market is getting more openned and this is good news for the customers.
http://www.youtube.com/watch?v=o_InjAmW5rI&feature=em-uploademail

How to interconnect two vrfs in the same switch Catalyst 6500

In a virtualized Data Centre environment, service is provided to customers with isolation in mind using vlans, vrfs and context based services in physical appliances such as firewalls, loadbalancers and Intrusion detection amd also as virtual appliances inside hypervisors.

When this level of virtualization is achieved, it is common that your requirement also increases and some impossible things are demanded. One of these situations are the interconnection of two vrfs in the same switch.

As you know, there can only be one layer3 vlan interface inside a 6500 chassis provided by the supervisor.

The following are solutions to interconnect two vrfs:

• Use an external firewall or router to route the traffic (bad option)
• Cross over cable in two ports (Most common)
• GRE connection between VRFs.
• Use RD for this requirement

The one that I find more stable is the crossover cable or fiber but it depends on a physical component which can fail.

The most clean solution is the GRE connection, but I have never tested in prodution.

Redistribute BGP to RIP

In order to redistribute BGP to RIP you will have to specify the metric of the routes when they arrive to rip engine otherwise they will have 255 distance and will not show up in the routing table. By default, the routes will be unreachable.

A examepl below:

router rip
 passive-interface default
 !
 address-family ipv4 vrf bank1
  redistribute bgp 64111 metric 1
  network 192.168.25.0
  neighbor 192.168.25.10
  no auto-summary
  version 2
 exit-address-family

10Gbps interface with Twin-ax cooper cable SFP-H10GB-CU3M

The twin ax cable SFP-H10GB-CU3M appear in a 3750 like this:

Te1/1/1   SERVER    connected    trunk        full    10G SFP-10GBase-CX1