Top-Ten-Workflow-Tips

 

Top 10 Wor orkf kflo low w Tips and Lo Log g Readi ding ng New York Yor k User Gro Grou u p Mee Meeti tin ng

Chr hris is Main Nor orth the east Consult Consu lta ant  Ap  A p r i l 20t h , 2005

1  

 Ag  A g en end da Log Lo g Re Reading di ng and Pe Perfor rf ormance mance Tunin un ing g nt erp rpret ret the S Se ess ssion ion Lo Log g #1 Inte #2 Tweak weak Sessio ss ion n Properti ro pertie es #3 Use Pa Part rtiti ition onin ing g #4 Consolid ons olida ate M Multi ultiple ple Look Lookups ups #5 Use Se Serv rve er Grids ri ds

2

 

 Ag  A g en end da

#6 Use Exp Expressi ression on Varia ri abl ble es #7 Use Changed-D Chang ed-Data ata Ca Cap pture #8 Ser ver Varia ri abl ble es #9 Redu duce ce Fa Fail ilur ure es #10 Use P Pe erf rfor ormance mance C Cou ount nte ers

3

 

Tip #1 #1: Int nte erp rpret ret th the e Sess ssio ion n Lo Log g • Session ssio n logs l ogs hold all sessio session n inform i nforma ation termin e bott bo ttlene leneck cks, s, error rr ors, s, loa lo ad sta st atist ti stic ics s • Determine

4

 

(#1) Ses s io ion n Log L ogs s —In it itiali ializ zat io ion n MASTER MAST ER> >C CMN MN_1 _168 688 8 All ocated [[12 1200 0000 0000 00]] b ytes ffrom rom pro cess memo ry ffor or [DTM Buf fer Pool ]. MASTER> MASTER > PE PETL_2 TL_240 4000 00 P Parallel arallel Pip elin eline e Engin e ini ti tializin alizing. g. MASTER MAST ER> > PE PETL_2 TL_240 4001 01 P Parallel arallel Pip elin eline e Engi Engine ne ru nni ng. ng . MASTER MAST ER> > PE PETL_2 TL_240 4003 03 IIni nitiali tiali zing sessi session on run . MAPPING> MAPPIN G> TM TM_60 _6014 14 Init ial ializi izing ng sess sessio ion n [S_M_C [S_M_CONN ONNECT ECTION ION_EX _EXTRACT TRACT]] at [ Thu Feb 05 16:35:41 20 2004] 04] MAPPING> TM_6101 Mapping name: M_CONNECTION_EXTRACT Version 1.0.0 MAPPING> CMN_1569 Server Mode: [ASCII] MAPPING> CMN_1570 Server Codepage: [MS Windows Latin 1 (ANSI), superset of Latin1] MAPPIN MAPP ING> G> T TM_ M_61 6151 51 S Sessio ession n Sort Order: [Binary] [B inary] MAPPIN MAPP ING> G> T TM_ M_61 6156 56 U Usin sin g L LOW OW precisi precision on decim al arith metic MAPPIN MAPP ING> G> TM TM_6 _618 180 0D Dea eadl dlock ock r etry lo logi gic c wi ll not be imp lemented. MAPPING> TE_7022 TShmWriter: Initialized MAPPIN MAPP ING> G> DBG_ DBG_21 2132 321 1 Loaded external m odu le li brary [[C:\P C:\Prog rog ram Files\Infor matic a\Infor a\Informatic matica a PowerCenterRT PowerCe nterRT 6 6.2 .2 - Server\ExtProc\crc Server\ExtProc\crc32 32.dll .dll]] MAPPING> MAPPIN G> TM_ TM_600 6007 7 DT DTM M ini ti alized s suc uccess cessfu fullll y fo r ses sessi sion on [S_M_C [S_M_CON ONNEC NECTIO TION_E N_EXTR XTRACT] ACT]

 A l l o c ated at ed m emo em o r y •  Al • Mappin pp ing g name

• Libraries 5

 

(#1) Sess ssio ion n Log L ogs—Loo s—Looku kups ps • Che Check ck Que Query ry • Check heck start ti time me heck number n umber of rrows ows • Check

ompare om pare amo moun untt o off d da ata • C with wit ha allo llotted tted memor memory y in session

6

TRANSF_1_1_1_1> DBG_21097 Default sql to create lookup look up cache: SELEC SELECT T CLASS_UID CLA SS_UID,CLA ,CLASS_ID SS_ID FR FROM OM IMW_C IMW_CLA LA SS ORDER ORDE R BY CL CLASS_ID,CLASS_UID ASS_ID,CLASS_UID

TRANSF_1_1 TRANSF_ 1_1_1_ _1_1> 1> DBG_2 DBG_2107 1079 9 Creati Creating ng Lo okup ok up Cache : (Thu Feb 05 16:35:41 2004)

TRANSF_1_1_1_1> DBG_21297 Lookup cache row co unt : 704

TRANSF_1_1_1_1> DBG_21294 Lookup cache creation completed : (Thu Feb 05 16:35:41 2004)

 

(#1) Ses s io ion n Log L ogs s —St at i s t ic ics s READER_1_1_1 READER_1_ 1_1]] Tot Total al R Run un Tim e = [1685. [1685.518 518988 988]] secs , Tot Total al Idle T Tim ime e = [1009. [1009.423 423504 504]] secs secs,, Bu Busy sy Percent age = [[40. 40.112 112006 006]. ]. TRANSF_1_1_1 TRANSF_1_ 1_1_1] _1] Tot Total al Run Tim Time e = [1524.4 [1524.4130 13078] 78] sec secs, s, Tot al Idl e Time = [1380. [1380.883 883920 920]] s secs, ecs, Busy Bu sy Pe Percent rcentage age = [[9. 9.41 4153 5372 72]. ]. TRANSF_1_1_1 TRANSF_1_ 1_1_2] _2] Tot Total al R Run un Tim e = [1705. [1705.480 480528 528]] secs secs,, Tot Total al Idle T Tim im e = [48.28 [48.28035 0356] 6] secs secs,, Bu Busy sy Percent age = [[97. 97.169 169105 105]. ]. WRITER WRI TER_1_ _1_1_1 1_1]] Tot Total al Run Time = [1660. [1660.035 035222 222]] secs secs,, Tot Total al Idl Idle e Tim Time e = [493.2 [493.2182 18257] 57] sec secs, s, Bu Busy sy Percent age = [[70. 70.288 288687 687]. ].

•

Busy Bus yp pe ercentage rcentages s ca c an iide dentif ntify y potentia p otentiall bottlene bott lenecks cks

 A d j u s t y o u r m ap app p i n g s t o l evel ev el o u t t h e bo b o t t l enec en eck ks •  Ad • Here the transforma transform ation is of bo bottl ttle eneck—consi neck—conside derr partitionin partiti oning g

•

Don’t on’ t need need 100 100% uti utili liz zati tion, on, but shou s hould ld b be e hi high gh and c clos lose e together 

7

 

(#1) Ses s io ion n Lo L o g s —Er r o r s • Find reje rejected cted recor records ds in the log file fi les s li mina ate erro errors, rs, especia especially lly if reocc reoccurr urring ing • Elimin

• Use e err rror or--handling handli ng stra st rategie tegies s

Keep Kee p you yourr session sessions s run runnin ning ge err rror or fr fre ee.  An  A n t i c i p at ate e er errr o r s an and d write m map app p i n g s t h at w i l l h an and dle these erro rrors rs iin n th the e appr ppropr opriate iate method. Each error tha t hatt occ occurs urs w will ill cause the session session to rrun un slower.

8

 

Ti p #2: Twea weak k Ses s io ion n Pr o pert perties ies • Experiment!! foll owing ng tip t ips s as as guideline gu idelines s • Use the followi (what what to t o llook ook for in Prope rop ertie rti es) iscove er settin settings gs optim o ptima al for f or your  y our  • Discov parti pa rticul cula ar system sy stem

 A n y s ett et t i n g s c h ang an g es m u s t c o n s i d er  •  An all other o ther setti settings ngs

9

 

(#2) Se Ser v er Ar Arc c h it ite ec t u r e—Mem o r y • DTM Buff Bu ffe er Poo Pooll Size Size sets amount amou nt of memor memory y forr Re fo Rea ader and and Writer ri ter buff bu ffe ers • Total number of blocks available • Optimal value is about 25MB • Block size of 64K means 25M/64K = 390 blocks

• Buf Buffe ferr Blo B lock ck Siz ize e sets sets size of blo blocks cks in t he pipe the pip eline lin e • Optimum size depends on the row size s ize being processed • 64KB

≈

• 128KB 10

64 rows of 1KB ≈

128 rows of 1KB

 

(#2) Ser v er Ar A r c h i t ec t u r e—DTM Par am et er s Sessi ssion on Task Parame rameters ters Control Cont rol th the e Pro Processin cessing g Pip ipe eli line ne and Are Ar e Foun ound d on the t he Prop rope erti rtie es and and Config Object Ta Tabs

11

 

(#2) Twea Tweak k in ing g Com omm m it In t er v als Commi om mitt Inte nt erva rv als Def ault 10, 0,0 000 ro rows ws targ targe et • Def • Can sw swit itch ch b be etw twe een target comm commit it and source sou rce comm commit it

•  A c o m m i t c an s l o w d o w n l o ad p r o c es ess s • La Large rge comm commit it leve levels ls ma may y ffill ill da data taba base se tra transaction nsaction logs log s • Small mall commits comm its ma may yb be e slowe slower  r 

12

 

(#2 (# 2) Tw Tw eak i n g Cac h e Si Si zes Cac Ca c h e Sizes faultt 1MB iind nde ex and 1MB da d ata • Defaul • Calc lcul ula ate how mu much ch c ca ache you w wil illl need need ttle le = Pagi ging ng • Too lilitt • Too mu much ch = Wast ste ed rre eso sourc urce es, wa wast ste ed ttim ime e et iin n either mappi mapping ng or se sessio ssion n (se (sessi ssion on overrides • S mapping)

• Powe ow erCe rCent nte er Se Serv rve er u uses ses cach cache e for Loo Lookup, kup, Aggrega Ag gregator tor,, Rank nk,, Joi Joiner ner and and Sor orter ter Transfo Transform rma ati tion ons s 13

 

Tip #3: Use Pa Part rtit itio ioni ning ng • When rre ead or o r wri w ritt es are at at 100% b usy us y da atabases tabases are part partit itio ioned ned • When d (use us e databa database se part partit itio ioni ning ng)) ransfor ormati mation ons s are at 100 100% busy bu sy • When ttransf

• When sou sourc rce e SQL is i s ttoo oo com c ompl ple ex • When a add ddit itio iona nall thr t hre ead wil w illl inc i ncrea rease se performance

14

 

(#3) Th r ead s , Par t i t i o n Po i n t s , an d St ag es

Shortcut_To_  ORDER_DATA (Flat File)

SQ_Shortcut  _To_ORDER_  DATA 1

FIL TRA NS

EXP_RPT

Reader Thread

Transformation Thread

(First Stage)

(Second Stage)

 AGG_RPT

Transformn Thread

Sh o r t c u t _To _    AGG_Cu stom  AGG_Cust om er   _SAL ES (Oracle)

Writer  Thread

(Third Stage) (Fourth Stage)

• Thr hre eads are crea created ted to mo move ve da data ta dow down n tthe he pip pipe eline li ne • The da data ta is mo moved ved in p pip ipe eli line ne sta stages ges de defi fined ned by partit pa rtition ion points

• By defa defaul ultt PowerC owerCe ent nte er assi ssigns gns a partiti parti tion on po poin intt Sourc ou rce e Qualifi uali fie er, T Ta arg rge et, Ag Aggr gre egato gator, r, a and nd Rank Transformations 15

at th the e

 

Term rmino inolog logy y in act action ion Pipeline

Th r ead 1

16

Th r ead 2

Thread Threa d3

 

17

 

No Tu Tu ni ning ng Sess ssii on Thr hre ead St St at s •

MASTER> P MASTER> PET ETL_240 L_24018 18 T Thr hrea ead d [READER [READER_1 _1_1 _1_1 _1]] crea cr eated ted fo forr t he read stage of partit p artitio ion np poi oint nt [[SQ SQ_O _ORP RPDT DTL] L] has comp co mpleted: leted: Total Run Time = Busy y [256.61 [256. 6199 9917 17]] sec secs, s, To Total tal Idl Idle e Tim Time e = [189 [189.6 .648 4826 261] 1] s secs ecs,, Bus Perr cen Pe centag tage e = [26 [ 26.0 .097 9760 606]. 6].

•

MASTER> P MASTER> PET ETL_240 L_24019 19 Thr Threa ead d [ TR TRANSF_ ANSF_1_ 1_1_ 1_1_ 1_1] 1] crea cr eated ted fo forr t he transformation st sta age of partition partit ion poi point nt [SQ_ [SQ_ORPDTL] has complete com pleted: d: Tot al Run Tim Total Time e = [255. [255.37 3745 4506 06]] s secs ecs,, T Tot otal al Idl Idle eT Tim ime e = [ 0.00 0.0000 0000 00]] s secs ecs,, Busy Perc rcent entage age = [100. [100.00 0000 0000 00]. ]. Bu sy Pe

•

MASTER> MASTE R> P PET ETL_240 L_24022 22 T Thr hrea ead d [WRIT [WRITER ER_1 _1_1 _1_1 _1]] crea cr eated ted fo forr th the e write stage of pleted: partit part itio ion n po poin t(s) [DEM [DEMAND2 _ALL_I _INS NSER ERTS TS, DW DWLDEM DISD SD]=] has com c ompl eted: Tot Total alint(s) Run Tim Time e AND2_ALL = [198 [198.2 .297 9746 469] 9] secs secs, , ,T Tot otal alLDEMDI Idle Time [173.083904] secs, Busy Percentage = [12.715021].

•

18

MASTER> PETL_24021 ***** END RUN INFO *****

 

Re/Par t it itii o n Poi Point nt Pipeline

Create Partition Point

PP

Th r ead 1

19

Th r ead 2

Thread Threa d3

 

Re/Partition Points Pipeline

PP

Th r ead 1

20

Th r ead 2

Thread Threa d3

 

Re/Partition Points Pipeline

PP

Threa Thread d1

21

Thr hrea ead d2

Thread Threa d4

 

Re/Partition Points Pipeline

PP

Thread d1 Threa

22

Thr hrea ead d2

Thread Threa d3

Thread Threa d4

 

 Ad  A d d i n g Par Partt i t i o n Po Poii n t s

23

 

24

 

25

 

Thr hre ead Stats Stats aft afte er add addin ing g Pa Part rtit itio ion n Poi Point nts s •

MASTER> MASTE R> PET PETL_2401 L_24018 8 Thread [ READER_ READER_1_1 1_1_1] _1] creat c reated ed ffor or the th e rea read d stage st age o f part itio partit ion n poi p oint nt [SQ_OR [SQ_ORPD PDTL] TL] has h as compl co mpleted: eted: Total Tot al Run Time = [160.6 [160.635 3586 861] 1] secs,, Total Idle secs Id le Time Tim e = [92.6 [92.679 7905 054] 4] secs, s ecs, Busy Bu sy Pe Perc rcentage entage = [42. [42.30 3048 4879 79]] .

•

MASTER> PET MASTER> PETL_2401 L_24019 9 Thread [ TRANSF_ TRANSF_1_1 1_1_1_ _1_1] 1] created c reated f or the th e transformation stage of partition partit ion point poi nt [SQ_O [SQ_OR RPDT PDTL] has completed: comp leted: Total Total Run Time = [158.843670] secs, Total Idle Time = [146.467909] secs, Busy Percentage = [7.791158].

•

MASTER> PET MASTER> PETL_2401 L_24019 9 Thread [ TRANSF_ TRANSF_1_1 1_1_1_ _1_2] 2] created c reated f or the th e transformation stage of partition point [exp_Evaluate_Conditions] has co mplet comp leted: ed: Total Run Time Tim e = [159.00 [159.0060 6092 92]] secs, secs , Tot Total al Idle Idl e Tim Time e = [15.0 [15.034 3449 492] 2] secs,, Busy secs Bu sy Pe Perc rcentage entage = [90 [ 90.5 .544 4470 707]. 7].

•

MASTER> PET MASTER> PETL_2401 L_24019 9 Thread [ TRANSF_ TRANSF_1_1 1_1_1_ _1_3] 3] created c reated f or the th e transformation stage of partiti p artition on poin p ointt [exp_Ta [exp_Target_Place rget_Placehol holder] der] has co mplet comp leted: ed: Total Run Time Tim e = [126.34 [126.3470 7018 18]] secs, secs , Tot Total al Idle Idl e Tim Time e = [72.7 [72.726 2616 160] 0] secs, Busy Bu sy Pe Perc rcentage entage = [42 [ 42.4 .439 3935 354]. 4].

•

MASTER> PET MASTER> PETL_2402 L_24022 2 Thread Thr ead [WR [ WRITE ITER_ R_1_1 1_1_1] _1] created c reated f or t he write stage o f partit part itio ion n poi p oint nt(s) (s) [DEMAND [DEMAND2_ 2_AL ALL_IN L_INSE SERT RTS, S, DWLDEM DWLDEMDI DISD SD]] has h as comp co mplet leted: ed: Total Run Time = [11 [ 112. 2.09 0954 5416 16]] secs, secs , Total Idle Idl e Time = [86 [ 86.4 .447 4793 935] 5] secs, s ecs, Busy Percentage = [22.880044].

26  

Part rtit itio ioni ning ng (Data Part rtit itio ion) n) Pipeline Create (addPartition #2) PP

Threa Thread d 1 Th r ead 2

Th r ead 3

PP

Thread Threa d4

Thread Threa d5

27  

Part rtit itio ioni ning ng (Data Part rtit itio ion) n) Part rtit itio ion n1 Part rtit itio ion n2 PP

PP

Thread Thread 1 Threa Thread d2

Threa Thread d3

Thread Threa d4

Thread Threa d5

Thread Thread 6 Threa Thread d7

Thread Threa d8

Thread Threa d9

Threa Thr ead d 10

28  

Part rtit itio ioni ning ng (Data Part rtit itio ion) n) Part rtit itio ion n1

PP

Part rtit itio ion n2 Thread Thread 1 Threa Thread d2

Threa Thread d3 PP

Thread 6 Thread Thread Thread 7

Threa Thread d8

PP

Thread Threa d4

Thread Threa d5

Thread Threa d9

Threa Thr ead d 10

PP

29  

Part rtit itio ioni ning ng (Data Part rtit itio ion) n) Part rtit itio ion n1

PP

Thread Thread 1 Threa Thread d2

Part rtit itio ion n2 Thread Threa d3

PP

Thread Thread 6 Threa Thread d7

Thread d8 Threa

PP

Thread Threa d4

Thread Thread 5

PP

Thread Threa d9

Threa Thr ead d 10

30  

Part rtit itio ioni ning ng (Data Part rtit itio ion) n) Part rtit itio ion n1

PP

Thread 1 Threa Thread Thread d2

Thread d3 Threa

PP

Thread Threa d4

Part rtit itio ion n2

PP

PP

Thread Threa d5

31

Thread Threa d 6 Thread Thread 7

Thread Threa d8

 

 Ad  A d d Dat Data a Part Par t i t i o n

Thread Threa d9

Threa Thr ead d 10

32  

Thr hre ead Stat Stat s af t er addi add i ng Dat a Par t i t io ion n •

***** RUN INFO FOR TGT LOAD ORDER GROUP [1], SRC PIPELINE [1] *****

•

MASTER> PETL_2 MASTER> PETL_2401 4018 8 Thread [READER_ [READER_1_1 1_1_1 _1]] c reated for the read stage st age of part it ition ion poi nt [ SQ_ SQ_ORP ORPDTL] DTL] has c omp leted: To tal Run Time = [147.332109] secs, Total Idle Time = [121.743271] secs, Busy Percentage = [17.368134].

•

MASTER> PETL_24019 Thread [TRANSF_1_1_1_1] created for the transformation stage of partition point [SQ_ORPDTL] has com pleted: Total Ru n Tim e = [145.68 [145.6833 3316] 16] secs, Tot al Idle Time = [ 14 140.5 0.503 03645 645]] s ecs, Bu sy Percentage = [3.55543 [3.555432]. 2].

•

MASTER> PETL_24019 Thread [TRANSF_1_1_1_2] created for the transformation stage of partition point [exp_Evaluate_Condi tio ns] has co mpl eted: Total Tot al Run Time = [146.01 [146.01755 7557] 7] secs, Total Tot al Idle Time = [25.919 [25.91962 622] 2] secs, Bu sy

•

•

•

Percentage = [82.248969]. MASTER> PETL_24019 Thread [TRANSF_1_1_1_3] created for the transformation stage of partition point [exp_Target_Placehold er] has c omp leted: Total Run Tim e = [83.09 [83.0968 6817] 17] secs, Total Idle Time = [36.874 [36.87484 846] 6] secs, Bu sy Percentage = [55.624238]. MASTER> PETL_2 MASTER> PETL_2402 4022 2 Thread [WRITER_1 [WRITER_1_1_ _1_1] 1] cr eated for the w rit e stage o f p arti tio n p oin t(s) [DEM [ DEMAND2_ AND2_ALL ALL _IN _INSER SERTS, TS, DWLDEMDI DWLDE MDISD] SD] has c omp leted: Total Run Time = [88.92735 [88.927355] 5] secs , Total Idle Time = [ 74 74.74 .7496 9625 25]] s ecs, Bu sy Percentage = [15.943047]. MASTER> MASTE R> PETL_2 PETL_2401 4018 8 Thread [READER_ [READER_1_1 1_1_2 _2]] c reated for the read stage st age of part it ition ion poi nt [ SQ_ SQ_ORP ORPDTL] DTL] has c omp leted: To tal Run Time = [106.301414] secs, Total Idle Time = [97.831194] secs, Busy Percentage = [7.968116].

•

MASTER> PETL_24019 Thread [TRANSF_1_1_2_1] created for the transformation stage of partition point [SQ_ORPDTL] has com pleted: Total Ru n Tim e = [103.61 [103.6150 5048] 48] secs, Tot al Idle Time = [ 10 102.7 2.736 36694 694]] s ecs, Bu sy Percentage = [0.84770 [0.847709]. 9].

•

MASTER> PETL_24019 Thread [TRANSF_1_1_2_2] created for the transformation stage of partition point [exp_Evaluate_Condi tio ns] has co mpl eted: Total Tot al Run Time = [145.72 [145.72891 8914] 4] secs, Total Tot al Idle Time = [27.767 [27.76789 894] 4] secs, Bu sy Percentage = [80.945515].

•

MASTER> PETL_24019 Thread [TRANSF_1_1_2_3] created for the transformation stage of partition point [exp_Target_Placehold er] has c omp leted: Total Run Tim e = [84.94 [84.9419 1995] 95] secs, Total Idle Time = [39.152 [39.15203 031] 1] secs, Bu sy Percentage = [53.907333].

•

MASTER> MASTE R> PETL_2 PETL_2402 4022 2 Thread [WRITER_1 [WRITER_1_1_ _1_2] 2] cr eated for the w rit e stage o f p arti tio n p oin t(s) [DEM [ DEMAND2_ AND2_ALL ALL _IN _INSER SERTS, TS, DWLDEMDISD] DWLDEMDI SD] has c omp leted: Total Run Time = [87.92420 [87.924204] 4] secs , Total Idle Time = [ 74 74.55 .5570 7008 08]] s ecs, Bu sy Percentage = [15.203091].

33  

(#3) Ad Addi ding ng Part rtit itio ions ns and Part rtit itio ion n Poi oint nts s

Shortcut_To_  ORDER_DATA (Flat File)

SQ_Shortcut  _To_ORDER_  DATA 1

FIL TRA NS

EXP_RPT

 AGG_RPT

Sh o r t c ustom t _om To _er    AGG_Cu  AGG_Cust  _SAL ES (Oracle)

Threads Thre ads - partiti partition on 1 Threads – partition partition 2 Threads – partition partition 3

3 Reader Threads (First Stage)

6 Transformation Threads (Second Stage)

(Third Stage)

3 Writer Threads (Fourth Stage)

 Ad d d i n g p ar artt i t i o n s i n c r eases eas es t h e n nu u m b er o f t h r ead eads s •  A

•  A  Ad d d i n g p ar artt i t i o n p o i n t s i n c r eases eas es t h e n nu u m b er  of pipe pi peli line ne sta stages ges

34  

(#3) Pa Par t it itii on Ty p es —Pas s Thr hrou oug gh • Define fin e your you r own parti partiti tions ons defi ne your you r own o wn SQL state s tatements ments • SQ define

• Partitio rti tions ns base based d on incomi inc oming ng data data values values • For cha ch ange ng eable bl e source sour ce queries • For “ evenly venly distrib di stribute uted” d” extract xtract data data • Whe hen n pa p artitio rti tions ns help help avoid voi d cont c onte ention/de nti on/dea adlocks dlo cks

35  

(#3) Pa Par t it itii on Ty p es —Pas s Thr hrou oug gh Sample Session with Pass Through Throu gh Partitioning

New Part rtit itio ion n Poi oint nt at the t he EXP_RPT Transformation

36  

(#3) Par t it itii on Ty p es —Rou oun n d Ro b in • Record co rds s alte lt erna rn ate betw betwe een partitio partit ions ns venly nly distri dis tribut bute ed be b etween tween the t he pa parti rtiti tions ons • Eve

• Use whe when n sou s ourc rce e reco record rds s are not no t evenl evenly y distri dis tribut bute ed or are are unpre unp redic dicta table ble

• Groupin rou ping g of o f da d ata is not nece necessary ssary amon among g the parti partitio tions ns

37  

(#3) Exampl xample e: Rou ound nd Rob obin in Part rtit itio ioni ning ng Source Pa Partitioning rtiti oning – Pass Through Target Pa Parti rtitio tionin ning g– Roun ound dR Robi obin n

Partitioned Flatt File Fla Fil e Order_Data_1.dat Order_Data_2.dat

38  

(#3) Par t i t i o n Ty p es —K ey Ran g e • Specif pecify y ke k eys and ranges ranges for f or key valu value es serr-de defi fine ned d distri dis tribut bution ion • Use

• Use when keys are defined defi ned ea easily si ly (numb nu mbe ers) rs ) • Com ompo poun und d ke k eys acce cc epte pt ed

39  

(#3) Ex am p l e: So u r c e an d Tar g et Part rtit itio ioni ning ng By Ke Key y Ra Rang nge e Sour ource ce Parti rtitio tionin ning g

Target Pa Parti rtitio tionin ning g

40

Leave the start range blank or the end range blank to force a < or >.

 

(#3) Par t i t i o n Type Typ es —Has h • Spe pecify cify keys keys on o n which wh ich to pa p artition rtit ion ser-defin define ed ke k eys to cre cr eate hash hash partitio partit ions ns • User-

• Use to ea easi sily ly dist di stri ribu bute te the data data eve evenl nly y • Data is grou gr ouped ped based based on keys keys • Has h auto-keys aut o-keys = Pow owe erCen rCen ter Serv Serve er uses u ses all groupe group ed or sorted sorted ports po rts as compoun co mpound d key key

41  

(#3) Ex am p l e: Has h Par t i t i o n

Hash pa partiti rtitioning oning used with an  Ag  A g g r eg egat ato or transformation

42  

(#3) Par t i t i o n Type Typ es —Dat ab as e • B as ed on o n Tar Tar g et IBM DB 2 Database ulti node loading loading • Multinode

• Targe rg et ta t abl ble e must mu st cont co nta ain a part partit itio ion n key • Non on--DB2 ta targ rge ets will wi ll default default to Pass Throu hr ough gh

43  

(#3) Part rtit itio ion n Poi oint nts s Sum umm m ary •  A s ess es s i o n p i p eli el i n e h has as Reader Read er,, Wri Wr i t er, er , and one o ne or mor m ore e Trans ransfo formatio rmation n thr t hre eads • Those ho se are default default pa p artit rt itio ion n poin po ints ts ty pe of pa p artit rt itio ioni ning ng ca c an be appli pp lie ed • One type at these points poi nts

ssi on Ma Mappings ppi ngs--Partiti rti tions ons”” ta tab b displays dis plays • The “ Session the parti partiti tion on type t ype all allowe owed d at at tha t hatt poin p ointt

44  

(#3) Partition Points Points Summa ummary ry Default fault p pa artitio rti tion n points poi nts

Read er

Tra ran n sf sfo o rma rmatt io io n

Tra ran n sf sfo o rrmation

Wri ritt er  er 

 Ad  A d d i n g p art ar t i t i o n p o i n t s

Read er

Tr an s f o r mation

Tr an s f o r m at i o n

Tr an s f o r mation

Wr i t er  

 A d d i n g an ad  Ad add d i t i o n al p ar artt i t i o n p o i n t m ay i n c r eas ease e session performance perfo rmance by adding additi additional onal threa threads ds

45  

Tip #4: Con onso soli lida date te Mul ulti tipl ple e Lo Look okup ups s • Loo Lookup kups s to look l ookups ups— —reduc reduce e and reuse onsolida idate te lookups look ups by using u sing lookup look up ove o verrides rrides • Consol

• Use re reusable usable lookup loo kups s whe w hen n multip mul tiple le call calls s are nee n eed d ed

• Use unconne unco nnecte cted d lookup l ookups s in i n condi c ondition tiona al functions

46  

(#4) Con onso soli lida dati ting ng Lo Look okup ups s

47  

(#4) Con onso soli lida dati ting ng Lo Look okup ups s

48  

(#4) Con onso soli lida dati ting ng Lo Look okup ups s

49  

Ti p #5: Us e Ser v er Gr i d s •  A g r i d o f s erv er v ers er s i n w h i c h s ess es s i o n s c an r u n  As s s i g n w o r k f l o w t o a mas m astt er s erv er v er t h at w i l l •  A run sessi sessions ons on the t he worke wor kerr servers servers urre ently ntl y Roun ound d Robi obin n • Curr

• Maste st er se s erve rv er ca c an de d etect whi w hich ch server  to run on

50  

(#5) Ser v er Gr i d Ti p s • Not a comp co mplete lete failov fail ove er sys s ystem— tem—M Maste st er se s erve rv er mus t always must always be up onfi figu gure re all all servers the t he same— same—para paramete meterr fil f ile es, • Con conne con necti ctions ons,, scripts, scrip ts, ema email il,, etc.

• Can incl in clud ude e diff di ffe erent rent s se erve rv er pla pl atfor tf orms ms sp ecifi ci fie ed se s erve rv er if if s specia peciall pro p roperties perties • Use spe are ne necessa cessary ry (co (conn nne ectio ct ions ns,, pr prox oxim imit ity y to sour so urce/ ce/targ targe ets, ts , hardwa hardw are, etc. tc .)

51  

Serv rve er Gri rid d Opt ptio ion n Load Balanc Balancin ing g and Fail Failov ove er for f or Heterogene terogeneous ous Sys ystems tems Off ff-grid -grid PowerCe PowerCent nter er Se Serv rver  er 

PowerCenter  Servers o on n Se Serv rver er G Gri rid d

Repository

52  

Serv rve er Gri rid d Opt ptio ion n Session3

Off ff-grid -grid PowerCe PowerCent nter er Se Serv rver  er 

Session2

PowerCenter  Servers o on n Se Serv rver er G Gri rid d

Session1 Sessi Se ssion5 on5 (off-grid)

Session4

Session6

Repository

53  

Ti p #6: Us e Ex p r es s io ion n Var iabl iable es • Uti tili liz ze variabl variable es in i n exp expre ress ssio ions ns to t o ca c aptur pt ure e previous previou s record’ record’s s data data us e with wit h filt f ilte ers to t o pe p erform rfo rm • Can use aggre ggr egate gate functi fun ctions ons

• Can use u se to co comp mpa are two tw o re r ecord co rds s (make make sure sur e sour so urce ce data data is sort so rte ed)

• ORDE ORDER R MATTERS MA TTERS!!

54  

(#6) Var i ab l es Ex am p l e

var_RUNNING_TOTAL + SALES_REVENUE

55  

Ti p #7: Us e Ch an g ed -Dat a Cap t u r e • Reduce du ce data volu vo lume mes s to t o change c hanges s onl o nly y du ce loa lo ad volu vo lume, me, incr in cre ease perf perfor orma manc nce e • Reduce

• May re r equir qu ire e spe sp ecia ci al log l ogic ic • Some soluti sol utions ons may may burden bu rden source sourc e syste sys tem m

56  

(#7) Us i n g Map Map p i n g Var Var i ab l es fo f o r CDC • Mappin pp ing g vari varia able bl es to cha ch ange ng e dates, dates, keys • Use mapping t he last change date/key rangevariables to store the • Max date/key stored in the repository • Successive runs, use the mapping variable in the SQ

Pros

Cons

• Only bring in affected records • Minimal impact to mappings

source to • Requires carry effective dates

• Restart or resetting dates possible

57  

(#7) Us i n g CRC Ch ec k s f o r CDC • CRC c h ecks eck s • Calculate CRC values for each record • Store CRC values in target tables • Join source and target tables where keys match but CRC values do not match

Pros

Cons

• Only changed or new records

• Overhead on source system • CRC function will slow mappings slightly

• Failure can just return

58  

(#7) Us in ing g EA I So l ut utio ion n f o r CDC EAI – Message bus message bus bu s tto o se s end only on ly cha ch ange ng ed or o r new • Use message records recor ds to t o Pow Powe erCent rCente er  l-ti time me capabil capabilit ities ies possi pos sibl ble e • Real-

• Minima ini mall impa i mpact ct on o n sourc so urce e syste sys tem m

59  

(#7) Usi sing ng PowerExch PowerExcha ang nge e CDC PowerExchange

• Pre re--buil bu iltt CDC CDC solu so luti tion ons s ffor or various variou s da d atabases tabases • Only proce pro cesses sses new new records or changed changed re r ecords cor ds • Low Low--leve levell logic lo gic provid pro vide es optim op tima al perfo performa rmance nce ini mall impa im pact ct on o n sourc so urce e syste sys tems ms • Minima • Supp upport orte ed by b y Informa Inform atica tic a (not you) you )

60  

Ti p #8: Ser v er Var i ab l es • Use S Se erve rv er Vari Varia able bl es to t o ensur ens ure e smoo sm ooth th mig rati migra tions ons betwee between n environ nvi ronme ments nts • Use V Va ariables riables in i n sessi sessions ons and workfl wor kflow ow obje obj ects pda ate usi us i ng PMREP o r Workf Wor kfll ow Manager  • Upd

61  

(# (#8 8) Se Ser v er Var i ab l es Ser v er Var i ab l e

Des c r i p t i o n

$PMRootDir 

 A root directory for use by server variables. (usu. PowerCenter Server installation directory)

$PMSessionLogDir 

Default directory for session logs. Defaults to $PMRootDir/SessLogs.

$PMBadFileDir 

Default directory for reject files. Defaults to $PMRootDir/BadFiles. Default directory for the index and data cache files. Defaults to

$PMCacheDir  $PMTargetFileDir 

$PMRootDir/Cache. Use a drive local to the PowerCenter Server. Default directory for target files. Defaults to $PMRootDir/TgtFiles.

$PMSourceFileDir 

Default directory for source files. Defaults to $PMRootDir/SrcFiles.

$PMExtProcDir 

Default directory for external procedures. (def.: $PMRootDir/ExtProc)

$PMTempDir 

Default directory for temporary files. Defaults to $PMRootDir/Temp.

$PMSuccessEmailUser 

Email address to receive post-session email when the session completes successfully.

$PMFailureEmailUser 

Email address to receive post-session email when the session fails.

62  

(# (#8 8) Se Ser v er Var i ab l es Ser v er Var i ab l e

Des c r i p t i o n

$PMSessionLogCount

Number of session logs the PowerCenter Server archives for the session. (Def: 0)

$PMSessionErrorThreshold

Number of non-fatal errors the PowerCenter Server allows before failing the session. Non-fatal errors include reader, writer, and DTM errors. If you want to stop the session on errors, enter the number of non-fatal errors you want to allow before stopping the session. The PowerCenter Server maintains an independent error count for each source, target, and transformation. Use to configure the Stop On option in the session properties.

$PMWorkflowLogDir 

Default directory for workflow logs. Defaults to $PMRootDir/WorkflowLogs.

$PMWorkflowLogCount

Number of workflow logs the PowerCenter P owerCenter Server archives for the workflow. Defaults to 0.

$PMLookupFileDir 

Default directory for lookup files. Defaults to $PMRootDir/LkpFiles. $ PMRootDir/LkpFiles.

63  

(# (#8 8) Se Ser v er Var i ab l es

64  

(#8) Server Variables • Sys ystem tem Varia ri able bl es: • $$$SessStartTime, SESSSTARTTIME, SYSDATE, WORKFLOWSTARTTIME

• User De Deff ine in ed Varia ri able bl es : • $DBConnection, $InputFile, $OutputFile, $PMFailureEmailUser, $PMSessionLogCount, $PMSessionLogDir, $PMSessionLogFile, $PMSuccessEmailUser, $PMWorkflowLogCount, $PMWorkflowLogDir, $Source, $Target

65  

Ti p #9: Red u c e Fa Fai l u r es Err rror or Hand ndli ling ng Stra tr ategie tegies s Caept pture ureeerr et rror ors s in into to tta abl ble es or fla fl at ffil ile es befor before e the th ey rea reach ch • th the targe targ

• Use for erro rrorr reports, corrections or warning warnings s to s sour ource ces s Bui lt-in E Built-in Error rror Ha Handl ndling ing Functi Functionality onality • 7.x allo allows ws b buil uiltt-in in error h ha andl ndling ing

• Captu pture res s funct fu nction ion errors Filter il ter Ea Earl rly! y!

• Downst own strea ream m rre ejecti jection on of bad data ca can n degrade degrade performance

66  

Ti p #10: Us e Per f o r m an c e Co u n t er s •  A  All l t r an ans s f o r m ati at i o n s h av ave ec co o u n t er ers s m ai ain n t ai ain n ed b y t h e server—use to iidentif server—use dentify y bot bottl tle enecks e ea arly rl y • Enable Counters for session via Collect Performance Data option • Server creates “session_name.perf” file for counter statistics • Default location is session log directory • Collecting performance data has some impact on session performance but not as much as tracing

• The imp import orta ant co count unte ers: • Reads and writes to disk (Aggregators, Ranks, Sorters Joiners) • Rows read from cache for Lookups • Efficiency counters

67  

(#10) En En ab li lin n g Per f o r m anc nce e Co Co un untt er s • Perfo rformance rmance cou counters nters pr provi ovide de a va variety riety of statisti statistics cs for each transfor transforma matio tion n iin n a ma map p

• Counters oun ters a are re enable nabled d in th the e session sessio n pr property operty

Sessio ession n Wiza Wizard rd:: Collect Performance Perfor mance D Data ata

68  

(#10) Vi ew i n g Per Per f o r m an c e Co u n t er s Right click on the Session Task while running, select Properties, then Performance tab

69  

Que uest stio ions ns and Ans Answe wers rs