MapReduce By example
Content
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
What is MapReduce?
MapReduce is a programming model fo processing large data sets with a parall distributed algorithm on a cluster [src: http://en. http://en.wikipedia.org/wiki/M wikipedia.org/wiki/MapReduce apReduce
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Hadoop is the open source implementati the model by Apache Software foundat The main pro*ect is composed by: + (-) + R0 + MapReduce 1ts ecosystem is composed by: + 2ig
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Hadoop 2.x
+ R0: R0: the resource manager manag er,, now called R0, R0, i detached from mapreduce framework
+ *ava packages are under org.apache.hadoop.map
0
127 views
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
MapReduce inspiration The name MapReduce comes from functional programming:
+ map is the name of a higher+order function function that applies a given to each element of a list. )ample in )cala:
val numbers = List(1,2,3,4,5) numbers.map(x => x * x) == List(1,4,9,16,25
+ reduce is reduce is the name of a higher+order function that analy4e a rec data structure and recombine through use of a given com operation the results of recursively recursively processing its consti building up a return value. )ample in )cala:
val numbers = List(1,2,3,4,5)
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
MapReduce by examples
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Oerall iew
Big Data Combined
0
127 views
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
How does Hadoop wor!?
"nit + adoop divides the input 9le stored on (-) into splits 7tipicall of an (-) block8 and assigns every split to a di'erent mapper, t assign every split to the mapper where the split physically reside
Mapper + locally, adoop reads the split of the mapper line by line + locally, adoop calls the method map7 8 of the mapper for every it as the key/value parameters + the mapper computes its application logic and emits other key/v
Shu#e and sort + locally, adoops partitioner divides the emitted output of the m partitions, each of those is sent to a di'erent reducer
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Simplied $ow %for deelopers&
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Seriali'able s Writable
+ )eriali4able stores the class name and the ob*ect representatio stream; other instances of the class are referred to by an handl class name: this approach is not usable with random access
+ -or the same reason, the sorting needed for the shu<e and so can not be used with )eriali4able
+ The deseriali4ation process creates a new instance of the ob*e adoop needs to reuse ob*ects to minimi4e computation
0
127 views
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Writable wrappers (aa primitie
Writable implementation
(aa class
Writable implementation
boolean
?oolean=ritable
)tring
Te5t
byte
?yte=ritable
byte[
?ytes=ritable
short
)hort=ritable
!b*ect
!b*ect=ritable
null
0ull=ritable
int
1nt=ritable @1nt=ritable
(aa collection
Writable implementation
Aoat
-loat=ritable
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
"mplementin) Writable* the Sum+ount clas public class #um$%unt implements &ritable$%mparable'#um$%unt>
%uble&ritable sum +nt&ritable %unt public #um$%unt() set(new %uble&ritable(), new +nt&ritable()) public #um$%unt(%uble sum, +nteer %unt) set(new %uble&ritable(sum), new +nt&ritable(%unt)) -
/0verride public v%id rite(ata0utput data0utput) throws +0xepti%n rite(data0utput)
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
,lossary -erm
Meanin)
&ob
The whole process to e5ecute: the input da the mapper and reducers e5ecution and the output data
Task
Cvery *ob is divided among the several mappers and reducers; a task is the *ob portion that goes to every single mapper an reducer
)plit
The input 9le is split into several splits 7the suggested si4e is the (-) block si4e, D$Mb8
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
MapReduce by examples
et/s start codin)0
Big Data Combined
0
127 views
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Word+ount
7the ello =orldF for MapReduce, available in adoop s
We want to count the occurrences of eery of a text le "nput 1ata* The te5t of the book G-latlandG ?y Cdwin bbott.
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Word+ount mapper public static class %7eni8erapper extends apper'0b:et, ext, ext, private final static +nt&ritable %ne = new +nt&ritable(1) private ext %rd = new ext()
/0verride public v%id map(0b:et 7e;, ext value, $%ntext %ntext) throws +0xepti%n, +nterruptedxepti%n
#trin%7eni8er itr = new #trin%7eni8er(value.t%#trin()) while (itr.as%re%7ens()) %rd.set(itr.next%7en().trim()) rite(%rd )
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Word+ount reducer
public static class +nt#um<eduer extends <eduer'ext,+nt&ritable,ext, private +nt&ritable result = new +nt&ritable()
/0verride public v%id redue(ext 7e;, +terable'+nt&ritable> values, $%nte throws +0xepti%n, +nterruptedxepti%n int sum = for (+nt&ritable val values) sum "= val.et() result set(sum)
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
MapReduce by examples
Word+ount
Results*
a ab abb%t abb%tt abbreviated abide abilit; able ablest ab%lised ab%liti%n ab%ut ab%ve
936 6 3 2 1 1 1 9 2 1 1 4 22
Hadoop Supplement
Big Data Combined
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
MapReduce testin) and debu))
+ MREnit is a testing framework based on &unit for u testing mappers, reducers, combiners 7well see late they are8 and the combination of the three + Mocking frameworks can be used to mock >onte5t other adoop ob*ects
+ Bocal&obRunner is a class included in adoop that l run a complete adoop environment locally, in a sing &@M, that can be attached to a debugger. Bocal&obR can run at most one reducer
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
MR3nit test for Word+ount
/est public v%id testapper() throws xepti%n new apriver'0b:et, ext, ext, +nt&ritable>() .itapper(new &%rd$%unt.%7eni8erapper()) .it+nput(ull&ritable.et(), new ext(?@%% bar @% .it0utput(new ext(?@%%?), new +nt&ritable(1)) .it0utput(new ext(?bar?), new +nt&ritable(1)) .it0utput(new ext(?@%%?), new +nt&ritable(1)) .runest() -
/est public v%id test<eduer() throws xepti%n List'+nt&ritable> @%%Aalues = new Brra;List'>() @%%Aalues.add(new +nt&ritable(1)) @%%Aalues.add(new +nt&ritable(1))
0
127 views
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
MR3nit test for Word+ount /est public v%id testap<edue() throws xepti%n
new ap<edueriver'0b:et, ext, ext, +nt&ritable, ext, +n .itapper(new &%rd$%unt.%7eni8erapper()) .it+nput(ull&ritable.et(), new ext(?@%% bar @%%? .it<eduer(new &%rd$%unt.+nt#um<eduer()) .it0utput(new ext(?bar?), new +nt&ritable(1)) .it0utput(new ext(?@%%?), new +nt&ritable(2)) .runest()
-
0
127 views
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
-op4
We want to nd the top5n used words of a te "nput 1ata* The te5t of the book G-latlandG ?y C. bbott. )ource: http://www.gutenberg.org/cache/epub/"#H/pg"#H.t5t
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
-op4 mapper
public static class %papper extends apper'0b:et, ext, ext, +nt&r private final static +nt&ritable %ne = new +nt&ritable(1) private ext %rd = new ext() private #trin t%7ens = ?C!DEF'>GGH=GGCGGIGG*GGGG,,.GGJ()KG?
/0verride public v%id map(0b:et 7e;, ext value, $%ntext %ntext) throws +0xepti%n, +nterruptedxepti%n
#trin leanLine = value.t%#trin().t%L%er$ase().replaeBll( #trin%7eni8er itr = new #trin%7eni8er(leanLine) while (itr.as%re%7ens()) %rd.set(itr.next%7en().trim())
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
-op4 reducer
public static class %p<eduer extends <eduer'ext, +nt&ritable, ext, +nt private ap'ext, +nt&ritable> %untap = new Nasap'>()
/0verride public v%id redue(ext 7e;, +terable'+nt&ritable> values, $%ntext throws +0xepti%n, +nterrup int sum = for (+nt&ritable val values) sum "= val.et() -
%untap.put(new ext(7e;), new +nt&ritable(sum))
-
/0verride protected v%id leanup($%ntext %ntext) throws +0xepti%n
+nterrup
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
MapReduce by examples
-op4
Results*
te %@ and t% a i in tat is ;%u m; it as
22P6 1634 19P 1PP 936 Q35 Q13 499 429 419 334 33 322
Hadoop Supplement
Big Data Combined
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
-op4
1n the shufe and sort phase, the partioner will s every single word 7the key8 with the value GHG to reducers.
ll these network transmissions can be minimi4e we reduce locally the data that the mapper will This is obtained by a Combiner
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
-op4 combiner
public static class $%mbiner extends <eduer'ext, +nt&ritable, ext, +
/0verride public v%id redue(ext 7e;, +terable'+nt&ritable> values, $%nt throws +0xepti%n, +nterruptedxepti%n int sum = for (+nt&ritable val values) sum "= val.et() %ntext.rite(7e;, new +nt&ritable(sum))
-
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
-op4 Hadoop output Without combiner ap input re%rds=4239 Map output records=37817 ap %utput b;tes=359621 +nput split b;tes=11P Combine input records=0 $%mbine %utput re%rds= <edue input r%ups=49PQ <edue su@@le b;tes=435261
With combiner
ap input re%rds=4239 Map output records=37817 ap %utput b;tes=359621 +nput split b;tes=116 Combine input records=37817 $%mbine %utput re%rds=2 <edue input r%ups=2 <edue su@@le b;tes=194
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
MapReduce by examples
+ombiners 1f the function computed is + commutatie [a J b K b J a + associatie [a J 7b J c8 K 7a J b8 J c we can reuse the reducer as a combiner0 Ma5 function works:
Big Data Combined
0
127 views
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
+ombiners Adanta)es of usin) combiners + 0etwork transmissions are minimi4ed 1isadanta)es of usin) combiners
+ adoop does not guarantee the e5ecution of a combi it can be e5ecuted #, H or multiple times on the same in
+ Ley+value pairs emitted from mapper are stored in loc
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
-op4 in5mapper combiner private ap'#trin, +nteer> %untap = new Nasap'>() private #trin t%7ens = ?C!DEF'>GGH=GGCGGIGG*GGGG,,.GGJ()K
/0verride public v%id map(0b:et 7e;, ext value, $%ntext %ntext) throws +0xepti%n, +nterruptedxepti%n
#trin leanLine = value.t%#trin().t%L%er$ase().replaeBll #trin%7eni8er itr = new #trin%7eni8er(leanLine) while (itr.as%re%7ens())
#trin %rd = itr.next%7en().trim() if (%untap.%ntainsRe;(%rd)) %untap.put(%rd, %untap.et(%rd)"1) else %untap put(%rd 1)
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
-op4 in5mapper reducer private ap'ext, +nt&ritable> %untap = new Nasap'>()
/0verride public v%id redue(ext 7e;, +terable'+nt&ritable> values, $%nte throws +0xepti%n, +nterruptedxepti%n int sum = for (+nt&ritable val values) sum "= val.et() -
%untap.put(new ext(7e;), new +nt&ritable(sum)) -
/0verride
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
+ombiners 5 output Without combiner
With combiner
ap input re%rds=4239 Map output records=37817 ap %utput b;tes=359621 +nput split b;tes=11P Combine input records=0 $%mbine %utput re%rds= <edue input r%ups=49PQ <edue su@@le b;tes=435261 educe input records=37817 <edue %utput re%rds=2
ap input re%rds=4239 Map output records=37817 ap %utput b;tes=359621 +nput split b;tes=116 Combine input records=378 $%mbine %utput re%rds=2 <edue input r%ups=2 <edue su@@le b;tes=194 educe input records=!0 <edue %utput re%rds=2
With in"mapper
With in"mapper and combin
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Mean
We want to nd the mean max temperature for ee "nput 1ata* Temperature in Milan 7((MM, M10, M8
112, J4., 5. 212, J5., 5.1 312, J5., Q.Q S 2912213, 3., 9.
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Mean mapper private ap'#trin, List'%uble>> maxap = new Nasap'>()
/0verride public v%id map(0b:et 7e;, ext value, $%ntext %ntext) throws +0xepti%n, +nterruptedxepti #trinCI values = value.t%#trin().split((?,?)) if (values.lent = 3) return #trin date = valuesCBI ext m%nt = new ext(date.substrin(2)) %uble max = %uble.parse%uble(valuesCBTI) if (maxap.%ntainsRe;(m%nt)) maxap.put(m%nt, new Brra;List'%uble>()) maxap.et(m%nt).add(max)
-
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Mean mapper private ap'#trin, List'%uble>> maxap = new Nasap'>()
/0verride public v%id map(0b:et 7e;, ext value, $%ntext %ntext) throws +0xepti%n, +nterruptedxepti #trinCI values = value.t%#trin().split((?,?)) if (values.lent = 3) return #trin date = valuesCBI ext m%nt = new ext(date.substrin(2)) %uble max = %uble.parse%uble(valuesCBTI) if (maxap.%ntainsRe;(m%nt)) maxap.put(m%nt, new Brra;List'%uble>()) maxap.et(m%nt).add(max)
-
"s this co
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Mean
Sample input data* 112, 212, 312, 412, 512,
., ., ., ., .,
1. 2. 2. 4. 3.
Mapper 67* lines 78 2 Mapper 62* lines 98 :8 ; Mapper67* mean < %7=.= > 2=.=& 2 Mapper62* mean < %2.= > :.= > 9.=&
Reducer mean < %7;.= > 9.=& 2 < @. ut the correct mean is*
0
127 views
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
MapReduce by examples
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
Mean mapper
private ap'ext, List'%uble>> maxap = new Nasap'>()
/0verride public v%id map(0b:et 7e;, ext value, $%ntext %ntext) throws +0xepti%n, +nterruptedxepti%n #trinCI values = value.t%#trin().split((?,?)) if (values.lent = 3) return #trin date = valuesCBI ext m%nt = new ext(date.substrin(2)) %uble max = %uble.parse%uble(valuesCBTI) if (maxap.%ntainsRe;(m%nt)) maxap.put(m%nt, new Brra;List'%uble>()) maxap.et(m%nt).add(max)
-
-his is cor
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Mean reducer private ap'ext, #um$%unt> sum$%untap = new Nasap'>()
/0verride public v%id redue(ext 7e;, +terable'#um$%unt> values, $%nt throws +0xepti%n, +nterruptedxepti%n #um$%unt t%tal#um$%unt = new #um$%unt() for (#um$%unt sum$%unt values)
t%tal#um$%unt.add#um$%unt(sum$%unt) -
sum$%untap.put(new ext(7e;), t%tal#um$%unt) -
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Mean
Results*
2212 2213 221 2211 3213 321 3212 3211 223 224 225
Q.23Q6923Q69231 Q.2 Q.P51P51P51P51P52 9.QP5Q142P5Q142P6 1.Q419354P3PQ96P 13.133333333333333 1P.54P3PQ96QQ4192 13.Q419354P3PQ96P 9.2QP5Q142P5Q142P 1.41344P2Q5P621 9.14642P5Q142P5Q2
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
MapReduce by examples
Mean
Result*
Hadoop Supplement
Big Data Combined
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
(oin
We want to combine information from the users "nformation from the posts le %a oin& "nput 1ata 5 3sers le* DuserEptrEidD DreputationD D)oldD DsilerD Dbron'eD NH####D$#"N NHON N#N N#N N#N NH###""#P$N NDQ$N N$N NH"N N#N NH###HOS#N NSDN N#N NQN N$N
"nput 1ata 5 Fosts le*
DidD DtitleD Dta)namesD DauthorEidD DbodyD DnodeEtypeD DparentEidD DabsEparentEidD DaddedEat NQQPN N=hether pdf of Enit and omework is availableUN NcsH#H pdfN NH#####$ON NN NVuestionN NW0 N"#H"+#"+" #O:#P:#D.SOSHOHJ##N NHN N"QH"N N-eedback on udio XualityN NcsH#H production audioN NH####QDHN NYpZ=e are looking for the audio in our videos. Tell us what you think and try to be as YemZspeci9cY/emZ as possible.Y/pZN
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
(oin mapper
/0verride public v%id map(0b:et 7e;, ext value, $%ntext %ntext) throws +0xepti%n, +nterruptedxepti%n ile#plit @ile#plit = (ile#plit) %ntext.et+nput#plit() #trin @ilename = @ile#plit.etWat().etame() #trinCI @ields = value.t%#trin().split((?Gt?))
if (@ilename.eXuals(?@%rum!n%des!n%[email protected]? )) if (@ields.lent > 5) #trin aut%r+d = @ieldsC3I.substrin(1, @ieldsC3I.len #trin t;pe = @ieldsC5I.substrin(1, @ieldsC5I.lent if (t;pe.eXuals(?Xuesti%n?)) %ntext.rite(new ext(aut%r+d), %ne) -
0
127 views
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
(oin reducer
/0verride public v%id redue(ext 7e;, +terable'+nt&ritable> values, $%ntext % throws +0xepti%n, +nterruptedxepti%n int p%stsumber = int reputati%n = #trin aut%r+d = 7e;.t%#trin() for (+nt&ritable value values)
int intAalue = value.et() if (intAalue == 1) p%stsumber "" else reputati%n = intAalue J2
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
(oin
Results*
3SIRE"1 P153Q 111949 11545 162P 11194P 1629 1623 111945 1624 111944 1625 1626
RIF3-A-"O4 119 12 36 6 231 29 1 45Q 16Q 114 1 93
S+ORI 3 1 1 2 1 1 2 4 1 3 1 1
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
MapReduce by examples
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
(oin
R code to plot data*
users 'J read.sv(@ile=?partJrJ?,sep=MGtM, eader=) usersEA2Ci(usersEA2 > 1,)I 'J pl%t(usersEA2, usersEA3, xlab=?<eputati%n?, ;lab=?umber %@ p%sts?, p=
Result*
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
MapReduce by examples
Hadoop Big Data Training Online
J5means
Hadoop Supplement
Big Data Combined
We want to a))re)ate 21 points in clusters usi J5means al)orithm "nput 1ata* random set of points
2.2Q5 1.P6 2.915 J.1612 J1.26
.91QP 2.12 1.36Q9 .P4P1 J1.423
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
J5means al)orithm
Big Data Combined
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
J5means mapper
/0verride protected v%id setup($%ntext %ntext) throws +0xepti%n, +nterrupt Y<+CI aeiles = %ntext.et$aeiles() entr%ids = Ytils.read$entr%ids(aeilesCI.t%#trin()) -
/0verride public v%id map(0b:et 7e;, ext value, $%ntext %ntext) throws +0xepti%n, +nterrupted #trinCI x; = value.t%#trin().split(? ?) d%uble x = %uble.parseDouble(x;CI) d%uble ; = %uble.parseDouble(x;C1I) int index = d%uble ministane = %uble.BT!ABLY for (int : = : ' entr%ids.si8e() :"")
0
127 views
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
J5means reducer
public class Reans<eduer extends <eduer'+nt&ritable, ext, ext, +n
/0verride protected v%id redue(+nt&ritable 7e;, +terable'ext> values, $%nt throws +0xepti%n, +nterruptedxepti%n %uble mx = d %uble m; = d int %unter =
for (ext value values) #trinCI temp = value.t%#trin().split(? ?) mx "= %uble.parse%uble(tempCI) m; "= %uble.parse%uble(tempC1I) %unter "" -
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
MapReduce by examples
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
J5means drier 5 7
public static v%id main(#trinCI ars) throws xepti%n
$%n@iurati%n %n@iurati%n = new $%n@iurati%n() #trinCI %terBrs = new Zeneri0pti%nsWarser(%n@iurati%n, ars).et<emaininB if (%terBrs.lent = 3) #;stem.err.println(?Ysae Reans 'in> '%ut> 'lusters!number>?) #;stem.exit(2) int entr%idsumber = +nteer.parse+nt(%terBrsC2I) %n@iurati%n.set+nt($%nstants.$<0+!YO<!B<Z, entr%idsumber) %n@iurati%n.set($%nstants.+WY!+L, %terBrsCI)
List'%ubleCI> entr%ids = Ytils.reate<and%m$entr%ids(entr%idsumber) #trin entr%idsile = Ytils.et%rmatted$entr%ids(entr%ids) Ytils.rite$entr%ids(%n@iurati%n, entr%idsile) b%%lean as$%nvered = false int iterati%n = do
%n@iurati%n.set($%nstants.0YWY!+L, %terBrsC1I " ?J? " iterati%n) if (laun[%b(%n@iurati%n)) #;stem exit(1)
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
MapReduce by examples
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
J5means drier 5 2
private static b%%lean laun[%b($%n@iurati%n %n@i)
[%b :%b = [%b.et+nstane(%n@i) :%b.set[%bame(?Reans?) :%b.set[arO;$lass(Reans.lass) :%b.setapper$lass(Reansapper.lass) :%b.set<eduer$lass(Reans<eduer.lass) :%b.setap0utputRe;$lass (+nt&ritable.lass) :%b.setap0utputAalue$lass (ext.lass) :%b.setum<edueas7s(1) :%b.add$aeile(new Wat($%nstants.$<0+#!+L).t%Yri())
ile+nput%rmat add+nputWat(:%b
Wat(%n@i et($%nstants +W
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
MapReduce by examples
J5means Results* 4.5Q 4.51Q9 4.19QP 5.235P 1.Q4Q 1.445 J.615 .Q1P 1.345 1 22Q2
.551 .612 1.5Q6 1.Q9P2 3.952 5.1P 4.Q5Q6 2.P32 3.955P 4 923P
2 2 2 2 1
Hadoop Supplement
Big Data Combined
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
Print
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Hints
+ Ese MapReduce only if you have really big data: )XB or are less e5pensive in terms of time needed to obtain the s results
+ Ese a lot of defensive checks: when we have a lot of data, want the computation to be stopped by a trivial 02C :+8 + Testing can save a lot of timeF
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
What is MapReduce?
MapReduce is a programming model fo processing large data sets with a parall distributed algorithm on a cluster [src: http://en. http://en.wikipedia.org/wiki/M wikipedia.org/wiki/MapReduce apReduce
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Hadoop is the open source implementati the model by Apache Software foundat The main pro*ect is composed by: + (-) + R0 + MapReduce 1ts ecosystem is composed by: + 2ig
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Hadoop 2.x
+ R0: R0: the resource manager manag er,, now called R0, R0, i detached from mapreduce framework
+ *ava packages are under org.apache.hadoop.map
0
127 views
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
MapReduce inspiration The name MapReduce comes from functional programming:
+ map is the name of a higher+order function function that applies a given to each element of a list. )ample in )cala:
val numbers = List(1,2,3,4,5) numbers.map(x => x * x) == List(1,4,9,16,25
+ reduce is reduce is the name of a higher+order function that analy4e a rec data structure and recombine through use of a given com operation the results of recursively recursively processing its consti building up a return value. )ample in )cala:
val numbers = List(1,2,3,4,5)
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
MapReduce by examples
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Oerall iew
Big Data Combined
0
127 views
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
How does Hadoop wor!?
"nit + adoop divides the input 9le stored on (-) into splits 7tipicall of an (-) block8 and assigns every split to a di'erent mapper, t assign every split to the mapper where the split physically reside
Mapper + locally, adoop reads the split of the mapper line by line + locally, adoop calls the method map7 8 of the mapper for every it as the key/value parameters + the mapper computes its application logic and emits other key/v
Shu#e and sort + locally, adoops partitioner divides the emitted output of the m partitions, each of those is sent to a di'erent reducer
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Simplied $ow %for deelopers&
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Seriali'able s Writable
+ )eriali4able stores the class name and the ob*ect representatio stream; other instances of the class are referred to by an handl class name: this approach is not usable with random access
+ -or the same reason, the sorting needed for the shu<e and so can not be used with )eriali4able
+ The deseriali4ation process creates a new instance of the ob*e adoop needs to reuse ob*ects to minimi4e computation
0
127 views
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Writable wrappers (aa primitie
Writable implementation
(aa class
Writable implementation
boolean
?oolean=ritable
)tring
Te5t
byte
?yte=ritable
byte[
?ytes=ritable
short
)hort=ritable
!b*ect
!b*ect=ritable
null
0ull=ritable
int
1nt=ritable @1nt=ritable
(aa collection
Writable implementation
Aoat
-loat=ritable
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
"mplementin) Writable* the Sum+ount clas public class #um$%unt implements &ritable$%mparable'#um$%unt>
%uble&ritable sum +nt&ritable %unt public #um$%unt() set(new %uble&ritable(), new +nt&ritable()) public #um$%unt(%uble sum, +nteer %unt) set(new %uble&ritable(sum), new +nt&ritable(%unt)) -
/0verride public v%id rite(ata0utput data0utput) throws +0xepti%n rite(data0utput)
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
,lossary -erm
Meanin)
&ob
The whole process to e5ecute: the input da the mapper and reducers e5ecution and the output data
Task
Cvery *ob is divided among the several mappers and reducers; a task is the *ob portion that goes to every single mapper an reducer
)plit
The input 9le is split into several splits 7the suggested si4e is the (-) block si4e, D$Mb8
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
MapReduce by examples
et/s start codin)0
Big Data Combined
0
127 views
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Word+ount
7the ello =orldF for MapReduce, available in adoop s
We want to count the occurrences of eery of a text le "nput 1ata* The te5t of the book G-latlandG ?y Cdwin bbott.
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Word+ount mapper public static class %7eni8erapper extends apper'0b:et, ext, ext, private final static +nt&ritable %ne = new +nt&ritable(1) private ext %rd = new ext()
/0verride public v%id map(0b:et 7e;, ext value, $%ntext %ntext) throws +0xepti%n, +nterruptedxepti%n
#trin%7eni8er itr = new #trin%7eni8er(value.t%#trin()) while (itr.as%re%7ens()) %rd.set(itr.next%7en().trim()) rite(%rd )
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Word+ount reducer
public static class +nt#um<eduer extends <eduer'ext,+nt&ritable,ext, private +nt&ritable result = new +nt&ritable()
/0verride public v%id redue(ext 7e;, +terable'+nt&ritable> values, $%nte throws +0xepti%n, +nterruptedxepti%n int sum = for (+nt&ritable val values) sum "= val.et() result set(sum)
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
MapReduce by examples
Word+ount
Results*
a ab abb%t abb%tt abbreviated abide abilit; able ablest ab%lised ab%liti%n ab%ut ab%ve
936 6 3 2 1 1 1 9 2 1 1 4 22
Hadoop Supplement
Big Data Combined
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
MapReduce testin) and debu))
+ MREnit is a testing framework based on &unit for u testing mappers, reducers, combiners 7well see late they are8 and the combination of the three + Mocking frameworks can be used to mock >onte5t other adoop ob*ects
+ Bocal&obRunner is a class included in adoop that l run a complete adoop environment locally, in a sing &@M, that can be attached to a debugger. Bocal&obR can run at most one reducer
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
MR3nit test for Word+ount
/est public v%id testapper() throws xepti%n new apriver'0b:et, ext, ext, +nt&ritable>() .itapper(new &%rd$%unt.%7eni8erapper()) .it+nput(ull&ritable.et(), new ext(?@%% bar @% .it0utput(new ext(?@%%?), new +nt&ritable(1)) .it0utput(new ext(?bar?), new +nt&ritable(1)) .it0utput(new ext(?@%%?), new +nt&ritable(1)) .runest() -
/est public v%id test<eduer() throws xepti%n List'+nt&ritable> @%%Aalues = new Brra;List'>() @%%Aalues.add(new +nt&ritable(1)) @%%Aalues.add(new +nt&ritable(1))
0
127 views
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
MR3nit test for Word+ount /est public v%id testap<edue() throws xepti%n
new ap<edueriver'0b:et, ext, ext, +nt&ritable, ext, +n .itapper(new &%rd$%unt.%7eni8erapper()) .it+nput(ull&ritable.et(), new ext(?@%% bar @%%? .it<eduer(new &%rd$%unt.+nt#um<eduer()) .it0utput(new ext(?bar?), new +nt&ritable(1)) .it0utput(new ext(?@%%?), new +nt&ritable(2)) .runest()
-
0
127 views
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
-op4
We want to nd the top5n used words of a te "nput 1ata* The te5t of the book G-latlandG ?y C. bbott. )ource: http://www.gutenberg.org/cache/epub/"#H/pg"#H.t5t
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
-op4 mapper
public static class %papper extends apper'0b:et, ext, ext, +nt&r private final static +nt&ritable %ne = new +nt&ritable(1) private ext %rd = new ext() private #trin t%7ens = ?C!DEF'>GGH=GGCGGIGG*GGGG,,.GGJ()KG?
/0verride public v%id map(0b:et 7e;, ext value, $%ntext %ntext) throws +0xepti%n, +nterruptedxepti%n
#trin leanLine = value.t%#trin().t%L%er$ase().replaeBll( #trin%7eni8er itr = new #trin%7eni8er(leanLine) while (itr.as%re%7ens()) %rd.set(itr.next%7en().trim())
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
-op4 reducer
public static class %p<eduer extends <eduer'ext, +nt&ritable, ext, +nt private ap'ext, +nt&ritable> %untap = new Nasap'>()
/0verride public v%id redue(ext 7e;, +terable'+nt&ritable> values, $%ntext throws +0xepti%n, +nterrup int sum = for (+nt&ritable val values) sum "= val.et() -
%untap.put(new ext(7e;), new +nt&ritable(sum))
-
/0verride protected v%id leanup($%ntext %ntext) throws +0xepti%n
+nterrup
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
MapReduce by examples
-op4
Results*
te %@ and t% a i in tat is ;%u m; it as
22P6 1634 19P 1PP 936 Q35 Q13 499 429 419 334 33 322
Hadoop Supplement
Big Data Combined
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
-op4
1n the shufe and sort phase, the partioner will s every single word 7the key8 with the value GHG to reducers.
ll these network transmissions can be minimi4e we reduce locally the data that the mapper will This is obtained by a Combiner
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
-op4 combiner
public static class $%mbiner extends <eduer'ext, +nt&ritable, ext, +
/0verride public v%id redue(ext 7e;, +terable'+nt&ritable> values, $%nt throws +0xepti%n, +nterruptedxepti%n int sum = for (+nt&ritable val values) sum "= val.et() %ntext.rite(7e;, new +nt&ritable(sum))
-
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
-op4 Hadoop output Without combiner ap input re%rds=4239 Map output records=37817 ap %utput b;tes=359621 +nput split b;tes=11P Combine input records=0 $%mbine %utput re%rds= <edue input r%ups=49PQ <edue su@@le b;tes=435261
With combiner
ap input re%rds=4239 Map output records=37817 ap %utput b;tes=359621 +nput split b;tes=116 Combine input records=37817 $%mbine %utput re%rds=2 <edue input r%ups=2 <edue su@@le b;tes=194
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
MapReduce by examples
+ombiners 1f the function computed is + commutatie [a J b K b J a + associatie [a J 7b J c8 K 7a J b8 J c we can reuse the reducer as a combiner0 Ma5 function works:
Big Data Combined
0
127 views
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
+ombiners Adanta)es of usin) combiners + 0etwork transmissions are minimi4ed 1isadanta)es of usin) combiners
+ adoop does not guarantee the e5ecution of a combi it can be e5ecuted #, H or multiple times on the same in
+ Ley+value pairs emitted from mapper are stored in loc
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
-op4 in5mapper combiner private ap'#trin, +nteer> %untap = new Nasap'>() private #trin t%7ens = ?C!DEF'>GGH=GGCGGIGG*GGGG,,.GGJ()K
/0verride public v%id map(0b:et 7e;, ext value, $%ntext %ntext) throws +0xepti%n, +nterruptedxepti%n
#trin leanLine = value.t%#trin().t%L%er$ase().replaeBll #trin%7eni8er itr = new #trin%7eni8er(leanLine) while (itr.as%re%7ens())
#trin %rd = itr.next%7en().trim() if (%untap.%ntainsRe;(%rd)) %untap.put(%rd, %untap.et(%rd)"1) else %untap put(%rd 1)
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
-op4 in5mapper reducer private ap'ext, +nt&ritable> %untap = new Nasap'>()
/0verride public v%id redue(ext 7e;, +terable'+nt&ritable> values, $%nte throws +0xepti%n, +nterruptedxepti%n int sum = for (+nt&ritable val values) sum "= val.et() -
%untap.put(new ext(7e;), new +nt&ritable(sum)) -
/0verride
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
+ombiners 5 output Without combiner
With combiner
ap input re%rds=4239 Map output records=37817 ap %utput b;tes=359621 +nput split b;tes=11P Combine input records=0 $%mbine %utput re%rds= <edue input r%ups=49PQ <edue su@@le b;tes=435261 educe input records=37817 <edue %utput re%rds=2
ap input re%rds=4239 Map output records=37817 ap %utput b;tes=359621 +nput split b;tes=116 Combine input records=378 $%mbine %utput re%rds=2 <edue input r%ups=2 <edue su@@le b;tes=194 educe input records=!0 <edue %utput re%rds=2
With in"mapper
With in"mapper and combin
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Mean
We want to nd the mean max temperature for ee "nput 1ata* Temperature in Milan 7((MM, M10, M8
112, J4., 5. 212, J5., 5.1 312, J5., Q.Q S 2912213, 3., 9.
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Mean mapper private ap'#trin, List'%uble>> maxap = new Nasap'>()
/0verride public v%id map(0b:et 7e;, ext value, $%ntext %ntext) throws +0xepti%n, +nterruptedxepti #trinCI values = value.t%#trin().split((?,?)) if (values.lent = 3) return #trin date = valuesCBI ext m%nt = new ext(date.substrin(2)) %uble max = %uble.parse%uble(valuesCBTI) if (maxap.%ntainsRe;(m%nt)) maxap.put(m%nt, new Brra;List'%uble>()) maxap.et(m%nt).add(max)
-
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Mean mapper private ap'#trin, List'%uble>> maxap = new Nasap'>()
/0verride public v%id map(0b:et 7e;, ext value, $%ntext %ntext) throws +0xepti%n, +nterruptedxepti #trinCI values = value.t%#trin().split((?,?)) if (values.lent = 3) return #trin date = valuesCBI ext m%nt = new ext(date.substrin(2)) %uble max = %uble.parse%uble(valuesCBTI) if (maxap.%ntainsRe;(m%nt)) maxap.put(m%nt, new Brra;List'%uble>()) maxap.et(m%nt).add(max)
-
"s this co
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Mean
Sample input data* 112, 212, 312, 412, 512,
., ., ., ., .,
1. 2. 2. 4. 3.
Mapper 67* lines 78 2 Mapper 62* lines 98 :8 ; Mapper67* mean < %7=.= > 2=.=& 2 Mapper62* mean < %2.= > :.= > 9.=&
Reducer mean < %7;.= > 9.=& 2 < @. ut the correct mean is*
0
127 views
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
MapReduce by examples
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
Mean mapper
private ap'ext, List'%uble>> maxap = new Nasap'>()
/0verride public v%id map(0b:et 7e;, ext value, $%ntext %ntext) throws +0xepti%n, +nterruptedxepti%n #trinCI values = value.t%#trin().split((?,?)) if (values.lent = 3) return #trin date = valuesCBI ext m%nt = new ext(date.substrin(2)) %uble max = %uble.parse%uble(valuesCBTI) if (maxap.%ntainsRe;(m%nt)) maxap.put(m%nt, new Brra;List'%uble>()) maxap.et(m%nt).add(max)
-
-his is cor
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Mean reducer private ap'ext, #um$%unt> sum$%untap = new Nasap'>()
/0verride public v%id redue(ext 7e;, +terable'#um$%unt> values, $%nt throws +0xepti%n, +nterruptedxepti%n #um$%unt t%tal#um$%unt = new #um$%unt() for (#um$%unt sum$%unt values)
t%tal#um$%unt.add#um$%unt(sum$%unt) -
sum$%untap.put(new ext(7e;), t%tal#um$%unt) -
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Mean
Results*
2212 2213 221 2211 3213 321 3212 3211 223 224 225
Q.23Q6923Q69231 Q.2 Q.P51P51P51P51P52 9.QP5Q142P5Q142P6 1.Q419354P3PQ96P 13.133333333333333 1P.54P3PQ96QQ4192 13.Q419354P3PQ96P 9.2QP5Q142P5Q142P 1.41344P2Q5P621 9.14642P5Q142P5Q2
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
MapReduce by examples
Mean
Result*
Hadoop Supplement
Big Data Combined
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
(oin
We want to combine information from the users "nformation from the posts le %a oin& "nput 1ata 5 3sers le* DuserEptrEidD DreputationD D)oldD DsilerD Dbron'eD NH####D$#"N NHON N#N N#N N#N NH###""#P$N NDQ$N N$N NH"N N#N NH###HOS#N NSDN N#N NQN N$N
"nput 1ata 5 Fosts le*
DidD DtitleD Dta)namesD DauthorEidD DbodyD DnodeEtypeD DparentEidD DabsEparentEidD DaddedEat NQQPN N=hether pdf of Enit and omework is availableUN NcsH#H pdfN NH#####$ON NN NVuestionN NW0 N"#H"+#"+" #O:#P:#D.SOSHOHJ##N NHN N"QH"N N-eedback on udio XualityN NcsH#H production audioN NH####QDHN NYpZ=e are looking for the audio in our videos. Tell us what you think and try to be as YemZspeci9cY/emZ as possible.Y/pZN
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
(oin mapper
/0verride public v%id map(0b:et 7e;, ext value, $%ntext %ntext) throws +0xepti%n, +nterruptedxepti%n ile#plit @ile#plit = (ile#plit) %ntext.et+nput#plit() #trin @ilename = @ile#plit.etWat().etame() #trinCI @ields = value.t%#trin().split((?Gt?))
if (@ilename.eXuals(?@%rum!n%des!n%[email protected]? )) if (@ields.lent > 5) #trin aut%r+d = @ieldsC3I.substrin(1, @ieldsC3I.len #trin t;pe = @ieldsC5I.substrin(1, @ieldsC5I.lent if (t;pe.eXuals(?Xuesti%n?)) %ntext.rite(new ext(aut%r+d), %ne) -
0
127 views
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
(oin reducer
/0verride public v%id redue(ext 7e;, +terable'+nt&ritable> values, $%ntext % throws +0xepti%n, +nterruptedxepti%n int p%stsumber = int reputati%n = #trin aut%r+d = 7e;.t%#trin() for (+nt&ritable value values)
int intAalue = value.et() if (intAalue == 1) p%stsumber "" else reputati%n = intAalue J2
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
(oin
Results*
3SIRE"1 P153Q 111949 11545 162P 11194P 1629 1623 111945 1624 111944 1625 1626
RIF3-A-"O4 119 12 36 6 231 29 1 45Q 16Q 114 1 93
S+ORI 3 1 1 2 1 1 2 4 1 3 1 1
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
MapReduce by examples
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
(oin
R code to plot data*
users 'J read.sv(@ile=?partJrJ?,sep=MGtM, eader=) usersEA2Ci(usersEA2 > 1,)I 'J pl%t(usersEA2, usersEA3, xlab=?<eputati%n?, ;lab=?umber %@ p%sts?, p=
Result*
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
MapReduce by examples
Hadoop Big Data Training Online
J5means
Hadoop Supplement
Big Data Combined
We want to a))re)ate 21 points in clusters usi J5means al)orithm "nput 1ata* random set of points
2.2Q5 1.P6 2.915 J.1612 J1.26
.91QP 2.12 1.36Q9 .P4P1 J1.423
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
J5means al)orithm
Big Data Combined
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
J5means mapper
/0verride protected v%id setup($%ntext %ntext) throws +0xepti%n, +nterrupt Y<+CI aeiles = %ntext.et$aeiles() entr%ids = Ytils.read$entr%ids(aeilesCI.t%#trin()) -
/0verride public v%id map(0b:et 7e;, ext value, $%ntext %ntext) throws +0xepti%n, +nterrupted #trinCI x; = value.t%#trin().split(? ?) d%uble x = %uble.parseDouble(x;CI) d%uble ; = %uble.parseDouble(x;C1I) int index = d%uble ministane = %uble.BT!ABLY for (int : = : ' entr%ids.si8e() :"")
0
127 views
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
J5means reducer
public class Reans<eduer extends <eduer'+nt&ritable, ext, ext, +n
/0verride protected v%id redue(+nt&ritable 7e;, +terable'ext> values, $%nt throws +0xepti%n, +nterruptedxepti%n %uble mx = d %uble m; = d int %unter =
for (ext value values) #trinCI temp = value.t%#trin().split(? ?) mx "= %uble.parse%uble(tempCI) m; "= %uble.parse%uble(tempC1I) %unter "" -
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
MapReduce by examples
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
J5means drier 5 7
public static v%id main(#trinCI ars) throws xepti%n
$%n@iurati%n %n@iurati%n = new $%n@iurati%n() #trinCI %terBrs = new Zeneri0pti%nsWarser(%n@iurati%n, ars).et<emaininB if (%terBrs.lent = 3) #;stem.err.println(?Ysae Reans 'in> '%ut> 'lusters!number>?) #;stem.exit(2) int entr%idsumber = +nteer.parse+nt(%terBrsC2I) %n@iurati%n.set+nt($%nstants.$<0+!YO<!B<Z, entr%idsumber) %n@iurati%n.set($%nstants.+WY!+L, %terBrsCI)
List'%ubleCI> entr%ids = Ytils.reate<and%m$entr%ids(entr%idsumber) #trin entr%idsile = Ytils.et%rmatted$entr%ids(entr%ids) Ytils.rite$entr%ids(%n@iurati%n, entr%idsile) b%%lean as$%nvered = false int iterati%n = do
%n@iurati%n.set($%nstants.0YWY!+L, %terBrsC1I " ?J? " iterati%n) if (laun[%b(%n@iurati%n)) #;stem exit(1)
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
MapReduce by examples
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
J5means drier 5 2
private static b%%lean laun[%b($%n@iurati%n %n@i)
[%b :%b = [%b.et+nstane(%n@i) :%b.set[%bame(?Reans?) :%b.set[arO;$lass(Reans.lass) :%b.setapper$lass(Reansapper.lass) :%b.set<eduer$lass(Reans<eduer.lass) :%b.setap0utputRe;$lass (+nt&ritable.lass) :%b.setap0utputAalue$lass (ext.lass) :%b.setum<edueas7s(1) :%b.add$aeile(new Wat($%nstants.$<0+#!+L).t%Yri())
ile+nput%rmat add+nputWat(:%b
Wat(%n@i et($%nstants +W
127 views
0
RELATED TITLES
0
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
MapReduce by examples
J5means Results* 4.5Q 4.51Q9 4.19QP 5.235P 1.Q4Q 1.445 J.615 .Q1P 1.345 1 22Q2
.551 .612 1.5Q6 1.Q9P2 3.952 5.1P 4.Q5Q6 2.P32 3.955P 4 923P
2 2 2 2 1
Hadoop Supplement
Big Data Combined
127 views
0
0
RELATED TITLES
MapReduce By example Uploaded by sumit04_28
Hadoop Full description
Save
Embed
Share
DEV 301 - Lab Guide
Hadoop Big Data Training Online
Hadoop Supplement
Big Data Combined
MapReduce by examples
Hints
+ Ese MapReduce only if you have really big data: )XB or are less e5pensive in terms of time needed to obtain the s results
+ Ese a lot of defensive checks: when we have a lot of data, want the computation to be stopped by a trivial 02C :+8 + Testing can save a lot of timeF
