1211 WP NetworkTroubleshooting

Published on May 2016 | Categories: Topics | Downloads: 37 | Comments: 0 | Views: 196
of 18
Download PDF   Embed   Report

1211 WP NetworkTroubleshooting

Comments

Content



!"#$%&' )&%*+,"-.%%#/01 203 4&%+,"5
63"0#/7/82#/%0
Brad Hale

!"#$% '( )'*+%*+,
Introduct|on ........................................................................................................................... 3
1roub|eshoot|ng Network Þerformance Issues........................................................................ 3
8ase||ne Network Þerformance........................................................................................................ 3
Co||ect Network Dev|ce Þerformance Metr|cs................................................................................... 3
Sw|tch]kouter CÞU Ut|||zat|on.......................................................................................................... 4
Sw|tch]kouter Memory Ut|||zat|on................................................................................................... 4
Interface]8andw|dth Ut|||zat|on....................................................................................................... S
1roub|eshoot|ng 8andw|dth and 1raff|c .................................................................................. 6
NetI|ow ........................................................................................................................................... 7
App||cat|ons ..................................................................................................................................... 9
Þrotoco|s.......................................................................................................................................... 9
1op 1a|kers .................................................................................................................................... 10
1roub|eshoot|ng Conf|gurat|on Issues................................................................................... 10
1roub|eshoot|ng IÞ Address Conf||cts.................................................................................... 12
Network 1roub|eshoot|ng Steps............................................................................................ 14
1oo|s for Network 1roub|eshoot|ng ...................................................................................... 1S
now So|arW|nds Can ne|p .................................................................................................... 1S
So|arW|nds Network Þerformance Mon|tor.................................................................................... 16
So|arW|nds NetI|ow 1raff|c Ana|yzer.............................................................................................. 16
So|arW|nds Network Conf|gurat|on Manager ................................................................................. 17
So|arW|nds IÞ Address Manager..................................................................................................... 17
So|arW|nds User Dev|ce 1racker ..................................................................................................... 18


-*+.'/01+2'*
1he only Lhlngs ln llfe LhaL are cerLaln are deaLh, Laxes and neLwork lssues. Ckay, l added Lhe lasL one
buL we all know LhaL no maLLer how carefully planned your neLwork deslgn ls, how much redundancy
you have bullL ln, or how much you proacLlvely monlLor your neLwork, you are bound Lo have a problem
aL some polnL. And when LhaL problem occurs, you need Lo Lhe sLeps and Lools Lo LroubleshooL Lhe
problem so you can mlnlmlze Lhe lmpacL Lo your users.
1hls paper wlll provlde some baslc guldance on LroubleshooLlng and Lhen look aL flve common ºneLwork
lssues" and provlde some baslc LroubleshooLlng and problem ldenLlflcaLlon Llps and Lools. 1hls paper
assumes LhaL Lhe reader ls famlllar wlLh baslc neLwork fundamenLals and proLocols. lf noL, please see
Lhe SolarWlnds WhlLe Þaper neLwork ManagemenL - 8ack Lo Lhe 8aslcs.
!.'0#$%,3''+2*4 5%+6'.7 8%.('.9"*1% -,,0%,
º1he neLwork ls slow Loday" ls wlLhouL a doubL one of Lhe mosL dlsllked phrases heard by neLwork
admlnlsLraLors. 1he neLwork has become a dumplng ground for problems LhaL orlglnaLe as ofLen as noL
from servers and appllcaLlons as from Lhe neLwork. 1hus, one of Lhe blggesL [obs of Lhe neLwork
admlnlsLraLor ls Lo defend Lhelr neLwork from belng labeled Lhe cause of Loday's problem. 8ecause slow
envlronmenL performance ls ofLen flrsL-and ofLen lncorrecLly- aLLrlbuLed Lo Lhe neLwork, rapld
ldenLlflcaLlon and problem lsolaLlon ls crlLlcal Lo Lhe admlnlsLraLor's workload.
:",%$2*% 5%+6'.7 8%.('.9"*1%
Popefully you have performed a basellne of your neLwork performance so you know Lhe normal worklng
condlLlons of your neLwork lnfrasLrucLure. 1hls basellne can Lhen be used for comparlson Lo caLch
changes LhaL could lndlcaLe a problem, provlde early lndlcaLors LhaL appllcaLlon and neLwork demands
are pushlng near Lhe avallable capaclLy, and allgn neLwork performance basellnes wlLh servlce-level
agreemenLs (SLAs).
lf you haven'L esLabllshed a basellne, Lhen you wlll need Lo rely on your equlpmenL vendors and Lhelr
recommended or ºbesL pracLlce" Lhresholds. ?ou can also use varlous neLwork equlpmenL or
monlLorlng forums Lo see whaL oLher l1 professlonals are dolng.
)'$$%1+ 5%+6'.7 ;%<21% 8%.('.9"*1% =%+.21,
neLwork devlce performance meLrlcs provlde lnformaLlon abouL Lhe sysLem resources on each
lndlvldual devlce. 1hese meLrlcs are crlLlcal ln ascerLalnlng wheLher a resource overuse problem ls a
cenLral cause of a reducLlon ln performance. CollecLlng and reporLlng on neLwork devlces helps Lhe
LroubleshooLlng admlnlsLraLor qulckly ldenLlfy wheLher Lhe devlce ls a source of Lhe problem or Lhe
problem lles wlLhln Lhe neLwork Lrafflc or appllcaLlon communlcaLlon lLself.
uevlce monlLorlng uslng Lhe Slmple neLwork ManagemenL ÞroLocol (SnMÞ) provldes a very devlce-
cenLrlc vlew of neLwork condlLlons. uslng SnMÞ, counLers on a devlce such as a rouLer, swlLch, or

flrewall can be measured and forwarded Lo a neLwork
managemenL sysLem for revlew. 1hls daLa ls useful for
undersLandlng performance condlLlons LhaL are
speclflc Lo LhaL devlce. Þerformance sLaLlsLlcs such as
CÞu uLlllzaLlon, lnLerface/8andwldLh uLlllzaLlon, and
Memory uLlllzaLlon represenL Lhe ma[orlLy of
performance lssues encounLered ln Lhe day-Lo-day
operaLlon of neLwork devlces. ?ou can monlLor Lhese
devlce sLaLlsLlcs uslng one of many commerclally
avallable neLwork monlLorlng sofLware producLs.

>62+13?@'0+%. )8A A+2$2B"+2'*
Common sympLoms of hlgh CÞu uLlllzaLlon wlLhln your swlLch or rouLer lnclude:
• Plgh percenLages ln Lhe show process cpu command ouLpuL
• lnpuL queue drops
• Slow performance
• Servlces such as 1elneL, console response, plng response, or updaLes fall
• Plgh buffer fallures
lf you are able Lo connecL Lo Lhe rouLer, Lhen you can use Lhe show process cpu (for Clsco rouLers)
command Lo check lf CÞu uLlllzaLlon ls hlgh due Lo lnLerrupLs or processes.
router#show processes
CPU utilization for five seconds: 0%/0%; one minute: 0%; five minutes: 0%
PID Q Ty PC Runtime(uS) Invoked uSecs Stacks TTY Process
1 C sp 602F3AF0 0 1627 0 2600/3000 0 Load Meter
2 L we 60C5BE00 4 136 29 5572/6000 0 CEF Scanner
3 L st 602D90F8 1676 837 2002 5740/6000 0 Check heaps
4 C we 602D08F8 0 1 0 5568/6000 0 Chunk Manager
5 C we 602DF0E8 0 1 0 5592/6000 0 Pool Manager
6 M st 60251E38 0 2 0 5560/6000 0 Timers
7 M we 600D4940 0 2 0 5568/6000 0 Serial Backgroun
8 M we 6034B718 0 1 0 2584/3000 0 OIR Handler
9 M we 603FA3C8 0 1 0 5612/6000 0 IPC Zone Manager
10 M we 603FA1A0 0 8124 0 5488/6000 0 IPC Periodic Tim
11 M we 603FA220 0 9 0 4884/6000 0 IPC Seat Manager
12 L we 60406818 124 2003 61 5300/6000 0 ARP Input
13 M we 60581638 0 1 0 5760/6000 0 HC Counter Timer
14 M we 605E3D00 0 2 0 5564/6000 0 DDR Timers
15 M we 605FC6B8 0 2 0 11568/12000 0 Dialer event

Clsco provldes Lwo greaL documenLs on 1roubleshooLlng Plgh CÞu uLlllzaLlon and 1roubleshooLlng Plgh
CÞu uLlllzaLlon Caused by lnLerrupLs.
>62+13?@'0+%. =%9'.C A+2$2B"+2'*
Memory ls a llmlLed resource on all neLwork devlces and musL be conLrolled and monlLored Lo ensure
LhaL uLlllzaLlon ls kepL ln check. A memory allocaLlon fallure means elLher Lhe neLwork devlce has used
I|gure 1: CÞU Load & Memory Ut|||zat|on from
So|arW|nds Network Þerformance Mon|tor

all avallable memory or Lhe memory has fragmenLed such LhaL Lhe devlce cannoL flnd a usable avallable
block.
lor Clsco rouLers, Lhe sympLoms of memory allocaLlon fallure lnclude, buL are noL llmlLed Lo:
• 1he console or log message: "°S?S-2-MALLCClAlL: Memory allocaLlon of 1028 byLes falled from
0x6013LC84, Þool Þrocessor, allgnmenL 0"
• 8efused 1elneL sesslons
• 1he show processor memory command ls dlsplayed no maLLer whaL command you Lype on a
console
• no ouLpuL from some show commands
• "Low on memory" messages
• 1he console message "unable Lo creaLe LxLC - no memory or Loo many processes"
• 8ouLer hanglng, no console response.

Þosslble causes of memory fallure lnclude:
ln Þrocessor Memory ("Þool Þrocessor" on all plaLforms)
• Memory Slze uoes noL SupporL Lhe Clsco lCS SofLware lmage
• Memory Leak 8ug
• Large CuanLlLy of Memory used for normal or Abnormal Þrocesses
• Memory lragmenLaLlon Þroblem or 8ug
• Memory AllocaLlon lallure aL Þrocess = <lnLerrupL level>

ln ÞackeL Memory
• noL Lnough Shared Memory for Lhe lnLerfaces
• 8uffer Leak 8ug
• 8ouLer 8unnlng Low on lasL Memory

lor addlLlonal deLall and LroubleshooLlng sLeps for Clsco rouLers, see 1roubleshooLlng Memory
Þroblems.

-*+%.("1%?:"*/62/+3 A+2$2B"+2'*
8efore you sLarL dlgglng lnLo Lhe gory deLalls of your
rouLer lnLerfaces, lL ls besL Lo slmply monlLor Lhe overall
bandwldLh uLlllzaLlon Lo deLermlne lf you even have a
problem. numerous open source or free Lools from
neLwork managemenL suppllers exlsL ln Lhe markeL LhaL
greaLly slmpllfy Lhe process of gaLherlng bandwldLh
uLlllzaLlon daLa and presenLlng lL ln an easy-Lo-consume
graphlcal formaL. SolarWlnds free 8eal-1lme 8andwldLh
Analyzer ls an example of a commerclally developed
free Lool LhaL dlsplays neLwork devlce lnLerface uLlllzaLlon.
I|gure 2: Interface Ut|||zat|on Us|ng So|arW|nds Iree
kea| 1|me 8andw|dth Mon|tor

lf you deLermlne LhaL you have a problem Lhen you wlll wanL Lo geL deLalled lnformaLlon abouL Lhe
lnLerface on your rouLer. Cn Clsco rouLers, you can vlew Lhe lnformaLlon abouL a parLlcular lnLerface
uslng Lhe ºshow lnLerface" command:
Router# show interfaces

Ethernet 0 is up, line protocol is up
Hardware is MCI Ethernet, address is 0000.0c00.750c (bia
0000.0c00.750c)
Internet address is 131.108.28.8, subnet mask is 255.255.255.0
MTU 1500 bytes, BW 10000 Kbit, DLY 100000 usec, rely 255/255, load
1/255
Encapsulation ARPA, loopback not set, keepalive set (10 sec)
ARP type: ARPA, ARP Timeout 4:00:00
Last input 0:00:00, output 0:00:00, output hang never
Last clearing of "show interface" counters 0:00:00
Output queue 0/40, 0 drops; input queue 0/75, 0 drops
Five minute input rate 0 bits/sec, 0 packets/sec
Five minute output rate 2000 bits/sec, 4 packets/sec
1127576 packets input, 447251251 bytes, 0 no buffer
Received 354125 broadcasts, 0 runts, 0 giants, 57186* throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
5332142 packets output, 496316039 bytes, 0 underruns
0 output errors, 432 collisions, 0 interface resets, 0 restarts
!.'0#$%,3''+2*4 :"*/62/+3 "*/ !."((21
8andwldLh monlLorlng and Lrafflc analysls are Lwo key acLlvlLles for every buslness envlronmenL.
Þerformlng each correcLly asslsLs Lhe neLwork admlnlsLraLor wlLh ldenLlfylng boLLlenecks. lL helps Lhe
admln ldenLlfy Lhe neLwork needs and uses of servers and Lhelr hosLed appllcaLlons, as well as how Lhe
neLwork needs of one l1 servlce lmpacLs anoLher. lL also dellvers hard daLa LhaL ob[ecLlvely verlfles Lhe
ablllLy of Lhe neLwork Lo meeL sLaLed Servlce Level AgreemenLs (SLAs).
1he Lwo mosL common ways ln whlch neLwork Lrafflc can be monlLored and measured for performance
are Lhrough packeL analysls and flow analysls. 1radlLlonal packeL-based monlLorlng Lools enable peerlng
lnLo lndlvldual packeLs Lo deLermlne Lhelr conLenLs, Lhe LransacLlons beLween sysLems, and Lhe deLalls
of communlcaLlons belng passed along LhaL neLwork. llow analysls provldes lnslghL lnLo Lhe flow of
Lrafflc wlLhln Lhe neLwork, speclflcally Lhe who and whaL of Lrafflc consumpLlon.
1he packeL-based approach ls a loL llke aLLempLlng Lo deLermlne Lhe cause of a Lrafflc [am by peeklng
lnLo each lndlvldual vehlcle. knowlng whaL people and cargo are Lravelllng wlLhln each vehlcle may be
helpful ln answerlng some quesLlons, buL lL's noL llkely Lo lllumlnaLe Lhe cause of Lhe sysLem-wlde
slowdown. llow analysls, on Lhe oLher hand, allows us Lo sLep back Lo see condlLlons on Lhe sysLem as a
whole. 1o help you undersLand Lhe dlfferences ln perspecLlve here, leL's Lake a look aL common ways
used Lo measure Lrafflc on a neLwork:

• Þrotoco| ana|yzers - ÞroLocol analyzers Lake a look aL neLwork condlLlons from Lhe perspecLlve
of Lhe packeL. 1hese Lools analyze conversaLlons beLween devlces on Lhe neLwork from Lhe
locaLlon where Lhe analyzer ls measurlng. 1hls lnformaLlon glves Lhe neLwork admlnlsLraLor an
exLremely deLalled vlew of lndlvldual LransacLlons beLween Lwo devlces and Lhe speclflc daLa
belng Lransferred beLween Lhem.
• nardware probes and d|str|buted ana|yzers - Pardware probes and dlsLrlbuLed analyzers are an
early aLLempL Lo overcome Lhe llmlLaLlons of an lndlvldual proLocol analyzer. 1hese Lools can be
poslLloned all across Lhe neLwork for Lhe gaLherlng of lnformaLlon. 1hey go far ln provldlng Lhe
whole-sysLem perspecLlve LhaL ls so dlfflculL Lo gaLher Lhrough Lhe prevlous Lwo perspecLlves.
• 1raff|c f|ow ana|yzers - 1hese Lools overcome Lhe admlnlsLraLlon headaches of hardware probes
and dlsLrlbuLed analyzers by leveraglng Lhe daLa flow capLure capablllLles of Lhe neLwork devlce
lLself. 1rafflc flow analyzers recelve flow daLa dlrecLly from monlLored devlces and analyze LhaL
daLa Lo galn Lhe hlgh-level perspecLlve needed for LroubleshooLlng lncldenLs across Lhe neLwork.
5%+D$'6
neLllow ls a neLwork Lrafflc monlLor proLocol developed by Clsco SysLems for collecLlng lÞ Lrafflc
lnformaLlon. Whlle Lhe Lerm neLllow has become a de-facLo lndusLry sLandard many oLher
manufacLurers supporL alLernaLlve flow Lechnologles lncludlng, !unlper (!flow), 3Com/PÞ, uell and
neLgear (s-flow), Puawel (neLSLream), AlcaLel-LucenL (Cflow), and Lrlcsson (8flow).
8ouLers and swlLches LhaL supporL neLllow collecL lÞ Lrafflc sLaLlsLlcs on all lnLerfaces where neLllow ls
enabled, and laLer exporL Lhose sLaLlsLlcs as neLllow records, Loward aL leasL one neLllow collecLor -
Lyplcally a server LhaL does Lhe acLual Lrafflc analysls. 1he neLllow collecLor Lhen processes Lhe daLa Lo
perform Lhe Lrafflc analysls and presenLaLlon ln a user-frlendly formaL. neLllow collecLors can Lake Lhe
form of hardware based collecLors or probes, or sofLware based collecLors. SolarWlnds neLllow 1rafflc
Analyzer (n1A) ls an example of a sofLware based neLllow collecLor LhaL collecLs Lrafflc daLa, correlaLes
lL lnLo a useable formaL, and Lhen presenLs lL Lo Lhe user ln a web based lnLerface.
MonlLorlng and analyzlng neLllow wlll help obLaln valuable lnformaLlon abouL neLwork users and
appllcaLlons, peak usage Llmes, and Lrafflc rouLlng. ln conLrasL wlLh LradlLlonal SnMÞ-dependenL
sysLems, neLllow-based Lrafflc monlLorlng has Lhe ablllLy Lo characLerlze Lrafflc from appllcaLlons and
users, undersLand Lhe Lrafflc paLLerns, provlde a hollsLlc vlew lnLo bandwldLh uLlllzaLlon and WAn Lrafflc,
supporL C8CoS valldaLlon and performance monlLorlng, be used for neLwork Lrafflc forenslcs, and ald ln
compllance reporLlng.
Conflgurlng neLllow on a Clsco rouLer ls a very sLralghLforward and easy process. ?ou can use a free
Lool such as SolarWlnds neLllow ConflguraLor or you can manually conflgure uslng Lhe followlng sLeps:
Command Þurpose
SLep 1
Router> enable
LnLers prlvlleged LxLC mode
LnLer your password lf prompLed
SLep 2
Router# configure terminal
LnLers global conflguraLlon mode

SLep 3
Router(config)# ip flow-export
Version 9
Lnables v9 daLa exporL for Lhe maln
cache
SLep 4
Router(config)# ip flow-export
templates refresh-rate 15
(CpLlonal) Speclfles Lhe refresh raLe ln
number of exporL packeLs. "#$%&'( ls an
lnLeger from 1 Lo 600. 1he defaulL ls 20
packeLs.
SLep 3
Router(config)# ip flow-export
template timeout-rate 90
(CpLlonal) Speclfles Lhe LlmeouL raLe ln
mlnuLes. )*+,'&( ls an lnLeger from 1 Lo
3600. 1he defaulL ls 30 mlnuLes
SLep 6
Router(config)# ip flow-export
template options export-stats
Speclfles Lhe opLlons LemplaLe exporL
sLaLlsLlcs, lncludlng how many exporL
packeLs have been senL and how many
flows have been exporLed.
SLep 7
Router(config)# ip flow-export
template options refresh-rate 25
(CpLlonal) Speclfles Lhe refresh raLe ln
number of exporL packeLs. "#$%&'( ls an
lnLeger from 1 Lo 600. 1he defaulL ls 20
packeLs.
SLep 8
Router(config)# ip flow-export
template options timeout-rate 120
(CpLlonal) Speclfles Lhe LlmeouL raLe ln
mlnuLes. )*+,'&( ls an lnLeger from 1 Lo
3600. 1he defaulL ls 30 mlnuLes.
SLep 9
Router(config)# end
Lnds Lhe conflguraLlon sesslon and
reLurns Lo prlvlleged LxLC mode

1o dlsplay Lhe sLaLlsLlcs from Lhe neLllow daLa exporL, lncludlng sLaLlsLlcs for Lhe maln cache and all
oLher enabled caches, use Lhe !"#$ &' ()#$ *+'#,- command ln user LxLC or prlvlleged LxLC mode. 1he
followlng ls sample ouLpuL from Lhe !"#$ &' ()#$ *+'#,- command:
Router# show ip flow export

Flow export is enabled
Exporting flows to 10.42.42.1 (9991) 10.0.101.254 (9991)
Exporting using source IP address 10.0.101.203
Version 5 flow records
Export Stats for 10.42.42.1 (9991)
3 flows exported in 3 udp datagrams
0 flows failed due to lack of export packet
3 export packets were sent up to process level
0 export packets were dropped due to no fib
0 export packets were dropped due to adjacency issues
0 export packets were dropped enqueuing for the RP
0 export packets were dropped due to IPC rate limiting
Export Stats for 10.0.101.254 (9991)
7 flows exported in 7 udp datagrams
0 flows failed due to lack of export packet
6 export packets were sent up to process level
0 export packets were dropped due to no fib
0 export packets were dropped due to adjacency issues
0 export packets were dropped enqueuing for the RP
0 export packets were dropped due to IPC rate limiting

1here are a number of commerclally avallable flow analysls and bandwldLh monlLor producLs LhaL
greaLly slmpllfy Lhe process of enabllng neLllow and Lhen dlsplay Lhe raw numbers lnLo easy-Lo-
lnLerpreL charLs and Lables.
LeL's Lake a look aL Lhree parLlcular use cases for uslng flow analysls for LroubleshooLlng bandwldLh and
Lrafflc.
EFF$21"+2'*,
When an appllcaLlon on Lhe neLwork beglns
consumlng more Lhan lLs falr share of neLwork
bandwldLh, lLs use wlll lmpacL Lhe capaclLy avallable
for oLher neLwork servlces. 1he problem wlLh
ldenLlfylng Lhese lncldenLs uslng oLher Lypes of
neLwork Lools ls LhaL Lhe reporLlng of problems Lends
Lo focus on Lhe neLwork servlce belng lmpacLed. lor
example, when Lhe problem occurs, Lhe neLwork
admlnlsLraLor usually sLarLs wlLh knowledge LhaL
AppllcaLlon 8 ºls slow Loday." 1he [ob ls Lhen Lhelrs Lo
deLermlne why Lhe servlce ls slow and whaL ls
lnhlblLlng lLs deslred level of performance. uslng
effecLlve flow analysls Lools, Lhe admlnlsLraLor can
easlly vlew Lhe Lrafflc and usage paLLerns across Lhe
enLlre neLwork Lo ldenLlfy LhaL AppllcaLlon A ls
acLually Lhe culprlL. Conversely, uslng Lools wlLh a
closer perspecLlve may lncorrecLly focus Lhe
admlnlsLraLor's LroubleshooLlng on AppllcaLlon 8,
whlle lgnorlng Lhe lmpacL of AppllcaLlon A.
8.'+'1'$,
A second and slmllar lssue occurs when a speclflc proLocol over consumes neLwork resources. SLreamlng
proLocols are an excellenL example of Lhls Lype of consLanL and predlcLable neLwork flow. When users
on a neLwork make use of sLreamlng appllcaLlons, Lhelr consumpLlon Lyplcally occurs aL a consLanL level
over an exLended perlod of Llme. ulfferenL Lhan LransacLlon-based proLocols, sLreamlng proLocols have
Lhe Lendency Lo saLuraLe avallable neLwork resources due Lo Lhe addlLlve effecL of mulLlple sLreams. Cne
user maklng use of one sLream may noL be llkely Lo cause a neLwork problem, buL 30 or 100 users
employlng an equal number of sLreams qulckly beglns saLuraLlng Lhe neLwork. unllke packeL-based Lools
LhaL analyze lndlvldual pleces as Lhey go by, flow analysls Lools enable Lhe ldenLlflcaLlon of Lhe source,
desLlnaLlon, and proLocol of sLreams across Lhe neLwork. 1he end resulL ls Lhe ablllLy Lo crafL effecLlve
neLwork pollcles LhaL enable sLreamlng proLocols where necessary whlle prevenLlng Lhose LhaL
negaLlvely lmpacL Lhe funcLlonallLy of Lhe neLwork.
I|gure 3: So|arW|nds NetI|ow 1raff|c Ana|yzer 1op 10
App||cat|ons kesource

!'F !"$7%.,
A flnal area for whlch flow analysls Lools are
parLlcularly well sulLed ls Lhe ldenLlflcaLlon of Lop
Lalkers or, who ls consumlng Lhe bandwldLh. 1he 1op
1alkers feaLure of neLllow can be useful for analyzlng
and LroubleshooLlng neLwork Lrafflc ln any one of Lhe
followlng ways: SecurlLy by vlewlng a llsL of Lhe Lop
Lalkers Lo see lf Lrafflc paLLerns are conslsLenL wlLh
uenlal of Servlce (uoS) aLLacks, Load balanclng
Lhrough Lhe ldenLlflcaLlon of Lhe mosL heavlly used
parLs of your neLwork, and general Lrafflc sLudy and
plannlng for your neLwork.





!.'0#$%,3''+2*4 )'*(240."+2'* -,,0%,
Cne of Lhe flrsL quesLlons neLwork admlnlsLraLors should ask Lhemselves when LroubleshooLlng ls ºuld
someLhlng on my neLwork change?" More Lhan 80° of neLwork lssues are Lhe resulL of devlce
conflguraLlon errors, many of whlch were unplanned, unauLhorlzed, or noL fully LesLed prlor Lo
deploymenL.
Popefully you have been keeplng an archlve of your devlce conflguraLlons so you can compare Lhe
currenL verslon Lo Lhe prevlously archlved verslons. lf you haven'L been, Lhen you need Lo sLarL
lmmedlaLely.
lor a Clsco rouLer, Lhe .,/"&0* /#1(&2 command allows you Lo save your lCS conflguraLlon ln Lhe
conflguraLlon archlve uslng a sLandard locaLlon and fllename preflx LhaL ls auLomaLlcally appended wlLh
an lncremenLal verslon number as each consecuLlve flle ls saved.
Router# configure terminal
Router(config)# archive
Router(config-archive)# path disk0:myconfig
I|gure 4: So|arW|nds NetI|ow 1raff|c Ana|yzer 1op
Þrotoco|s kesource


?ou Lhen save Lhe currenL runnlng conflguraLlon ln Lhe conflguraLlon archlve as follows:
Router# archive config

1he show arch|ve command dlsplays lnformaLlon on Lhe flles saved ln Lhe conflguraLlon archlve as
shown ln Lhe followlng sample ouLpuL:
Router# show archive

There are currently 1 archive configurations saved.
The next archive file will be named disk0:myconfig-2
Archive # Name
0
1 disk0:myconfig-1 <- Most Recent
2

Assumlng LhaL you have a conflg archlve, you can perform a llne-by-llne comparlson of any Lwo
conflguraLlon flles and generaLe a llsL of Lhe dlfferences beLween Lhem uslng Lhe !"#$ .,/"&0* /#1(&2
3&((*,*1/*! command.
show archive config differences[filename1(path)[filename2(path)][ignorecase]]
1he ouLpuL wlll dlsplay Lhe resulLs of Lhe dlff operaLlon performed on Lhe conflguraLlon flles. A plus
symbol (+) lndlcaLes LhaL Lhe conflguraLlon llne exlsLs ln -*.&+#)&/0"#'12 buL noL ln -*.&+#)&30"#'12
whlle a mlnus symbol (-) lndlcaLes LhaL Lhe conflguraLlon llne exlsLs ln -*.&+#)&30"#'12 buL noL ln
-*.&+#)&/0"#'124 An exclamaLlon polnL (!) wlLh descrlpLlve commenLs ls used Lo ldenLlfy order-senslLlve
conflguraLlon llnes whose locaLlon ls dlfferenL ln -*.&+#)&30"#'12 Lhan ln -*.&+#)&/0"#'12.
+ip subnet-zero
+ip name-server 10.4.4.4
+voice dnis-map 1
+dnis 111
interface Ethernet1/0
+no ip address
+shutdown
+ip default-gateway 10.5.5.5
+ip classless
+access-list 110 deny ip any host 10.1.1.1
+access-list 110 deny ip any host 10.1.1.2
+access-list 110 deny ip any host 10.1.1.3
+snmp-server community private RW
-no ip subnet-zero
interface Ethernet1/0
-ip address 10.7.7.7 255.0.0.0
-no ip classless
-snmp-server community public RO




As opposed Lo relylng on a cumbersome and hard Lo
declpher CLl LroubleshooLlng process, Lhe neLwork
admlnlsLraLor may wanL Lo conslder one of Lhe many
commerclally avallable neLwork change and
conflguraLlon managemenL Lools LhaL wlll auLomaLe
and slmpllfy Lhe process of managlng devlce
conflguraLlons.
Cnce you deLermlne LhaL a conflg has changed, you
can replace Lhe currenL runnlng conflg wlLh any saved
conflg flle uslng Lhe conf|gure rep|ace command. 1hls
funcLlonallLy can be used Lo reverL Lo a prevlous
conflguraLlon sLaLe, effecLlvely rolllng back any
conflguraLlon changes LhaL were made slnce Lhe
prevlous conflguraLlon sLaLe was saved.
Router# configure replace disk0:myconfig

This will apply all necessary additions and deletions
to replace the current running configuration with the
contents of the specified configuration file, which is
assumed to be a complete configuration, not a partial
configuration. Enter Y if you are sure you want to proceed. ? [no]: Y

Total number of passes: 1
Rollback Done
!.'0#$%,3''+2*4 -8 E//.%,, )'*($21+,
lÞ address confllcLs occur when Lwo devlces on a neLwork are asslgned Lhe same lÞ address resulLlng ln
one or boLh belng dlsabled and loslng connecLlvlLy unLll Lhe confllcL ls resolved. lÞ address confllcLs are
almosL always Lhe resulL of conflguraLlon errors lncludlng: asslgnmenL of Lhe same sLaLlc lÞ address by a
neLwork admlnlsLraLor, asslgnmenL of a sLaLlc lÞ address wlLhln Lhe uPCÞ range (dynamlc range)
resulLlng ln Lhe same address belng auLomaLlcally asslgned by Lhe local uPCÞ server, an error ln Lhe
uPCÞ server, or a sysLem comlng back onllne afLer an exLended perlod ln sLand-by or hlbernaLe mode
wlLh an lÞ address LhaL has been re-asslgned and ls ln use on Lhe neLwork.
Pere are a number of sLeps LhaL you can Lake Lo LroubleshooL Lhls pesky problem.
Step 1 - Look Ior Cver|app|ng IÞ Address kanges on ¥our DnCÞ Server
lf you are uslng mulLlple uPCÞ servers, you wlll flrsL wanL Lo verlfy LhaL no Lwo servers have overlapplng
lÞ address ranges. 1hls can be as slmple as comparlng Lhe lÞ address ranges and looklng for overlaps
I|gure S: So|arW|nds Network Conf|gurat|on Manager
Compare Conf|gs

when Lhe servers are uslng dynamlc or auLomaLlc allocaLlon of lÞ addresses. lf Lhey are uslng sLaLlc
allocaLlon, Lhen you wlll need Lo revlew each hard coded lÞ address asslgnmenL.
Step 2 - Look for Dup||cate Stat|c IÞ Addresses
Look for devlces on Lhe neLwork segmenL LhaL have been sLaLlcally conflgured wlLh Lhe dupllcaLe lÞ
address. Cnce found, you can elLher reconflgure Lhe devlce Lo use uPCÞ or you can conflgure Lhe uPCÞ
server Lo sLop asslgnlng Lhe dupllcaLed lÞ address.
Step 3 - I|nd the Conf||ct|ng MAC Addresses
lf sLeps 1 and 2 do noL produce resulLs, you wlll need Lo flnd Lhe MAC addresses of Lhe confllcLlng
devlces. Slnce Lhe MAC address ls unlque for each devlce on Lhe neLwork, you can look for devlces LhaL
conLaln Lhe same lÞ address buL wlLh dlfferenL MAC addresses. ?ou can use Lhe Address 8esoluLlon
ÞroLocol (A8Þ) Lo esLabllsh a correspondence beLween Lhe lÞ address and Lhe MAC address. SLarL aL
your core rouLer and use Lhe !"#$ &' .,' command:
Router# show ip arp

Protocol Address Age(min) Hardware Addr Type Interface
Internet 172.16.233.229 - 0000.0c59.f892 ARPA Ethernet0/0
Internet 172.16.233.218 - 0000.0c07.ac00 ARPA Ethernet0/0
Internet 172.16.233.19 - 0000.0c63.1300 ARPA Ethernet0/0
Internet 172.16.233.309 - 0000.0c36.6965 ARPA Ethernet0/0
Internet 172.16.168.11 - 0000.0c63.1300 ARPA Ethernet0/0
Internet 172.16.168.254 9 0000.0c36.6965 ARPA Ethernet0/0
lf you were Lo see Lwo lÞ addresses wlLh dlfferlng hardware addresses Lhen you have locaLed your
problem devlces.
Step 4 - 1race the Locat|on of the Dev|ce
Þerhaps you wanL Lo know Lhe physlcal locaLlon or aL leasL Lhe swlLch porL LhaL Lhe offendlng devlces are
connecLed Lo. Cne way ls Lo go Lo Lhe swlLch and use Lhe !"#$ 4./5.33,*!! -.6)* command. 1hls wlll
show you Lhe MAC address for each porL.
switch# show mac-address-table
Mac Address Table
-------------------------------------------

Vlan Mac Address Type Ports
---- ----------- -------- -----
1 0007.e9e2.2d7d DYNAMIC Fa0/5
1 0009.0f30.07e9 DYNAMIC Fa0/48
1 0009.5bbc.af04 DYNAMIC Fa0/28
1 00e0.bb2c.30d1 DYNAMIC Gi0/1
1 00e0.bb2c.3e5f DYNAMIC Gi0/1
Total Mac Addresses for this criterion: 5

Switch#

unforLunaLely, you need Lo run Lhls command from each swlLch and, lf Lhe neLwork ls down, you wlll
have Lo go Lo Lhe console of each swlLch. 1hls can be very Ledlous and Llme consumlng noL Lo menLlon
loglsLlcally challenged ln Lhe case of geographlcally dlsLrlbuLed neLworks.
AnoLher alLernaLlve ls Lhe use of commerclally avallable
swlLch porL managemenL Lools LhaL wlll Lrace Lhe
locaLlon of a devlce on a neLwork auLomaLlcally.
SolarWlnds user uevlce 1racker ls a devlce Lracklng and
swlLch porL managemenL Lool LhaL qulckly locaLes a
devlce on Lhe neLwork by searchlng on Lhe lÞ address,
PosLname or MAC address.
Þrevent|ng Conf||cts |n the Iuture
Cnce you have ldenLlfled and correcLed lÞ address confllcLs, here are some Llps Lo prevenL fuLure
confllcLs:
• use uPCÞ Lo reduce Lhe chances of manually asslgnlng dupllcaLe addresses.
• SeL your uPCÞ server Lo deLecL lÞ address confllcLs.
• Modlfy Lhe uPCÞ lease duraLlon Lo someLhlng less Lhan Lhe defaulL lease Llme of 8 days.
• use mulLlple uPCÞ servers, each havlng lL own speclflc scope.
• 8eserve lÞ addresses lnsLead of asslgnlng sLaLlc lÞ addresses.
• use auLomaLed uPCÞ, unS, and lÞ address managemenL and monlLorlng Lools
Lven by followlng Lhese Llps, Lhere sLlll remalns Lhe posslblllLy LhaL lÞ confllcLs wlll occur. l would
encourage you Lo evaluaLe a commerclally avallable lÞ address managemenL producL LhaL allows you Lo
cenLrally manage, monlLor, alerL, and reporL on your lÞ lnfrasLrucLure. 8y proacLlvely managlng and
monlLorlng your lÞ address space you can slgnlflcanLly reduce Lhe chances of lÞ address confllcLs.
5%+6'.7 !.'0#$%,3''+2*4 >+%F,
1here are llLerally hundreds of neLwork LroubleshooLlng flow charLs avallable on Lhe lnLerneL Loday and
we are noL abouL Lo prescrlbe one over Lhe oLher. Powever, you wlll flnd LhaL ln all Lhese flow charLs,
successful LroubleshooLlng relles on loglc and meLhodology and follow Lhese baslc sLeps:
1. ldenLlfy Lhe sympLoms - documenL Lhe sympLoms
2. ldenLlfy Lhe scope of Lhe problem - geographlc, demographlc, or chronologlcal
3. ueLermlne lf anyLhlng has changed on Lhe neLwork - has Lhere been a hardware or sofLware
change?
4. ueLermlne Lhe mosL probable cause of Lhe problem - no, lL's noL always Lhe user
3. lmplemenL soluLlon
6. 1esL Lhe soluLlon
So|arW|nds User Dev|ce 1racker Lndpo|nt Deta||s

7. uocumenL Lhe soluLlon
And whlle noL speclflcally called ouL ln Lhese sLeps, lL ls lmporLanL Lo remember Lo pay aLLenLlon Lo Lhe
obvlous and don'L dlscounL Lhe slmple quesLlons. 1o puL lL blunLly, don'L forgeL Lo check Lhe cables.
!''$, ('. 5%+6'.7 !.'0#$%,3''+2*4
As we have shown LhroughouL Lhe paper, Lhere are llLerally hundreds of open source, free, or
commerclally llcensed producLs avallable Lo monlLor and LroubleshooL neLwork performance, Lrafflc and
bandwldLh, conflguraLlons, and lÞ lnfrasLrucLure. 8elow are some guldellnes on plcklng Lhe rlghL Lool for
your needs.
• MulLlple vendor devlce supporL - lL would be very dlfflculL ln Lhls day and age Lo flnd a neLwork
LhaL conslsLs of equlpmenL from a slngle vendor. Whlle all vendors provlde some Lype of Lool or
uLlllLy LhaL wlll manage and monlLor Lhelr own equlpmenL, lL ls crlLlcal LhaL you look for a Lool
LhaL allows you Lo monlLor all of your dlfferenL vendors ln a slngle pane of glass.
• SupporL for mulLlple sLandard proLocols lncludlng: SnMÞ, lCMÞ, and Syslog for neLwork
managemenL, 8uÞ, WMl, and WS -ManagemenL for Wlndows managemenL, and neLllow, !-
llow, sllow, lÞllx, and neLSLream for flow based Lrafflc monlLorlng.
• 8eal-Llme and hlsLorlcal analysls capablllLles. AlLhough mosL problems ln neLwork admlnlsLraLlon
dlrecLly relaLe Lo how Lhe neLwork operaLes rlghL now, Lhe only effecLlve way Lo ascerLaln
Loday's behavlors ls Lo vlew Lhem ln comparlson wlLh yesLerday's or lasL week's.
• vlsuallzaLlons accesslble from anywhere. As a neLwork admlnlsLraLor, you're noL always slLLlng ln
your offlce. Þroblems and lssues Lend Lo pop up all across Lhe neLwork, some of whlch requlre
on-slLe supporL. ln Lhese cases, havlng vlsuallzaLlons LhaL can be accessed from anywhere-for
example, uslng a sLandard Web browser-glves you Lhe ablllLy Lo Lake your LoolseL Lo wherever
Lhe problem exlsLs.
• urlll-down supporL. WlLh drlll-down supporL lL ls posslble Lo qulckly move from Lhe hlghesL-level
vlew down lnLo speclflc problems as needed. urlll-down supporL reduces on-screen cluLLer,
enabllng a slngle-gllmpse and hlgh-level vlew durlng perlods of nomlnal acLlvlLy.
• AffordablllLy. LasLly, any LoolseL used ln LroubleshooLlng and resolvlng lssues musL cosL less Lhan
Lhe amounL of beneflL lL provldes. Lxpenslve soluLlons Lake longer Lo pay for Lhemselves and
may be more dlfflculL Lo obLaln ln a Llme of shrlnklng l1 budgeLs. llndlng Lhe Lool LhaL meeLs
your needs aL an accepLable cosL ls lmporLanL Lo galnlng Lhe blggesL reLurn on your lnvesLmenL.
G'6 >'$".H2*/, )"* G%$F
SolarWlnds award-wlnnlng neLwork managemenL sofLware makes lL easy Lo dlscover and map neLwork
devlces, monlLor neLwork performance, analyze neLwork Lrafflc, manage and back up neLwork
conflguraLlons, Lrack lÞ addresses, flnd rogue devlces, and much more.



>'$".H2*/, 5%+6'.7 8%.('.9"*1%
='*2+'.
SolarWlnds neLwork Þerformance MonlLor (nÞM)
makes lL easy Lo qulckly deLecL, dlagnose, and resolve
performance lssues and dellvers real-Llme vlews and
dashboards LhaL enable you Lo vlsually Lrack neLwork
performance aL a glance. Þlus, uslng dynamlc
neLwork Lopology maps and auLomaLed neLwork
dlscovery, you can deploy and keep up wlLh your
evolvlng neLwork
• Slmpllfles deLecLlon, dlagnosls, & resoluLlon
of neLwork lssues - before ouLages occur
• 1racks response Llme, avallablllLy, & upLlme
of rouLers, swlLches, & oLher SnMÞ-enabled
devlces
• Shows performance sLaLlsLlcs ln real Llme vla
dynamlc, drlllable neLwork maps
• lncludes ouL-of-Lhe-box dashboards, alerLs, reporLs, & experL guldance on whaL Lo monlLor &
how
• AuLomaLlcally dlscovers SnMÞ-enabled neLwork devlces & Lyplcally deploys ln less Lhan an hour
>'$".H2*/, 5%+D$'6 !."((21 E*"$CB%.
SolarWlnds neLllow 1rafflc Analyzer (n1A) enables
you Lo capLure daLa from conLlnuous sLreams of
neLwork Lrafflc and converL Lhose raw numbers lnLo
easy-Lo-lnLerpreL charLs and Lables LhaL quanLlfy
exacLly how Lhe corporaLe neLwork ls belng used, by
whom and for whaL purpose.
• MonlLors neLwork bandwldLh & Lrafflc
paLLerns down Lo Lhe lnLerface level
• ldenLlfles whlch users, appllcaLlons, &
proLocols are consumlng Lhe mosL bandwldLh
• PlghllghLs Lhe lÞ addresses of Lop Lalkers
• Analyzes Clscoº neLllow, !unlperº !-llow,
lÞllx, sllowº, & Puawel neLSLream`

I|gure 6: So|arW|nds Network Þerformance Mon|tor's Summary
Þage
I|gure 7: So|arW|nds NetI|ow 1raff|c Ana|yzer Summary Þage


>'$".H2*/, 5%+6'.7 )'*(240."+2'* ="*"4%.
SolarWlnds neLwork ConflguraLlon Manager (nCM)
keeps you ahead of neLwork lssues wlLh lmmedlaLe
vlslblllLy lnLo Lhe cause and effecL relaLlonshlp
beLween conflguraLlon errors and neLwork
performance. Þlus, you can resL easy and save Llme
wlLh feaLures such as nlghLly conflg backups, bulk
conflg changes, user Lracklng, and lnvenLory and
compllance reporLlng.
- Lnables bulk conflguraLlon, communlLy sLrlng,
ACL, & MAC address changes
- AuLomaLes neLwork conflguraLlon backups &
compllance reporLlng
- ueLecLs & reporLs on conflguraLlon pollcy
vlolaLlons & dellvers real-Llme alerLs
- ÞroLecL agalnsL unauLhorlzed, unscheduled, or
erroneous conflg changes
- AuLomaLlcally dlscovers SnMÞ-enabled
neLwork devlces & Lyplcally deploys ln less Lhan an hour

>'$".H2*/, -8 E//.%,, ="*"4%.
SolarWlnds lÞ Address Manager (lÞAM) enables you
and your Leam Lo dlLch your spreadsheeLs for an easy-
Lo-use, cenLrallzed lÞ address monlLorlng and
managemenL soluLlon. now lL's easler Lhan ever Lo
manage MlcrosofLº uPCÞ servlces, monlLor MlcrosofL
unS and Clscoº uPCÞ servers, and manage your lÞ
address space, all from an lnLulLlve, cenLrallzed Web
console.
- CenLrally manage, alerL, & reporL on your lÞ
address space
- Manage & monlLor MlcrosofL uPCÞ/unS
servlces & monlLor Clsco uPCÞ servers
- uellvers role-based access & conLrol from an
lnLulLlve web based lnLerface
- AlerL noLlflcaLlons help prevenL your subneLs & uPCÞ
scopes from fllllng up
- AuLomaLlcally dlscovers used & unused addresses & Lyplcally deploys ln less Lhan an hour
I|gure 8: So|arW|nds Network Conf|gurat|on Manager Summary
Þage
I|gure 9: So|arW|nds IÞ Address Manager Summary Þage


>'$".H2*/, A,%. ;%<21% !."17%.
SolarWlnds user uevlce 1racker enables you Lo
qulckly flnd devlces on your neLwork, creaLe devlce
waLch llsLs, map swlLch porLs and Lrack swlLch
capaclLy.
• 1rack user and devlce locaLlons by MAC
address, lÞ address, or PosLname
• Map and monlLor swlLches by porLs used,
CÞu load, memory used and more
• 8ecelve lmmedlaLe alerLs when a speclfled
devlce connecLs Lo Lhe neLwork












SolarWinds (NYSE: SWI) provides powerful and affordable IT management software to customers worldwide -
from Fortune 500 enterprises to small businesses. The company works to put its users first and remove the
obstacles that have become “status quo” in traditional enterprise software. SolarWinds products are
downloadable, easy to use and maintain, and provide the power, scale, and flexibility needed to address users’
management priorities. SolarWinds online user community, hLLp://Lhwack.com, is a gathering-place where
tens of thousands of IT pros solve problems, share technology, and participate in product development for all
of the company’s products. Learn more
today at hLLp://solarwlnds.com.
I|gure 10: So|arW|nds User Dev|ce 1racker Summary Þage

Sponsor Documents

Or use your account on DocShare.tips

Hide

Forgot your password?

Or register your new account on DocShare.tips

Hide

Lost your password? Please enter your email address. You will receive a link to create a new password.

Back to log-in

Close