-
Notifications
You must be signed in to change notification settings - Fork 0
/
OUTPUT.txt
591 lines (591 loc) · 60.5 KB
/
OUTPUT.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/04/11 16:52:17 INFO SparkContext: Running Spark version 2.2.0
19/04/11 16:52:18 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:356)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:371)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:364)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:272)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:790)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2430)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2430)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2430)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:295)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at G14HM2.main(G14HM2.java:20)
19/04/11 16:52:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/04/11 16:52:18 INFO SparkContext: Submitted application: Homework
19/04/11 16:52:18 INFO SecurityManager: Changing view acls to: Raffaele_DNDM
19/04/11 16:52:18 INFO SecurityManager: Changing modify acls to: Raffaele_DNDM
19/04/11 16:52:18 INFO SecurityManager: Changing view acls groups to:
19/04/11 16:52:18 INFO SecurityManager: Changing modify acls groups to:
19/04/11 16:52:18 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(Raffaele_DNDM); groups with view permissions: Set(); users with modify permissions: Set(Raffaele_DNDM); groups with modify permissions: Set()
19/04/11 16:52:18 INFO Utils: Successfully started service 'sparkDriver' on port 52142.
19/04/11 16:52:18 INFO SparkEnv: Registering MapOutputTracker
19/04/11 16:52:18 INFO SparkEnv: Registering BlockManagerMaster
19/04/11 16:52:18 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/04/11 16:52:18 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/04/11 16:52:18 INFO DiskBlockManager: Created local directory at C:\Users\Raffaele_DNDM\AppData\Local\Temp\blockmgr-1398b951-eae8-4402-9d51-e93c1dd4d6bc
19/04/11 16:52:18 INFO MemoryStore: MemoryStore started with capacity 1984.5 MB
19/04/11 16:52:18 INFO SparkEnv: Registering OutputCommitCoordinator
19/04/11 16:52:18 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/04/11 16:52:18 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.56.1:4040
19/04/11 16:52:18 INFO Executor: Starting executor ID driver on host localhost
19/04/11 16:52:18 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 52183.
19/04/11 16:52:18 INFO NettyBlockTransferService: Server created on 192.168.56.1:52183
19/04/11 16:52:18 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/04/11 16:52:18 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.56.1, 52183, None)
19/04/11 16:52:18 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.56.1:52183 with 1984.5 MB RAM, BlockManagerId(driver, 192.168.56.1, 52183, None)
19/04/11 16:52:18 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.56.1, 52183, None)
19/04/11 16:52:18 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.56.1, 52183, None)
19/04/11 16:52:19 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 232.0 KB, free 1984.3 MB)
19/04/11 16:52:19 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 22.6 KB, free 1984.3 MB)
19/04/11 16:52:19 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.56.1:52183 (size: 22.6 KB, free: 1984.5 MB)
19/04/11 16:52:19 INFO SparkContext: Created broadcast 0 from textFile at G14HM2.java:27
19/04/11 16:52:19 INFO FileInputFormat: Total input paths to process : 1
19/04/11 16:52:19 INFO SparkContext: Starting job: count at G14HM2.java:31
19/04/11 16:52:19 INFO DAGScheduler: Got job 0 (count at G14HM2.java:31) with 2 output partitions
19/04/11 16:52:19 INFO DAGScheduler: Final stage: ResultStage 0 (count at G14HM2.java:31)
19/04/11 16:52:19 INFO DAGScheduler: Parents of final stage: List()
19/04/11 16:52:19 INFO DAGScheduler: Missing parents: List()
19/04/11 16:52:19 INFO DAGScheduler: Submitting ResultStage 0 (text-sample.txt MapPartitionsRDD[1] at textFile at G14HM2.java:27), which has no missing parents
19/04/11 16:52:19 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 3.1 KB, free 1984.2 MB)
19/04/11 16:52:19 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1884.0 B, free 1984.2 MB)
19/04/11 16:52:19 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.56.1:52183 (size: 1884.0 B, free: 1984.5 MB)
19/04/11 16:52:19 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006
19/04/11 16:52:19 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (text-sample.txt MapPartitionsRDD[1] at textFile at G14HM2.java:27) (first 15 tasks are for partitions Vector(0, 1))
19/04/11 16:52:19 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
19/04/11 16:52:19 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 4865 bytes)
19/04/11 16:52:19 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, executor driver, partition 1, PROCESS_LOCAL, 4865 bytes)
19/04/11 16:52:19 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
19/04/11 16:52:19 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
19/04/11 16:52:19 INFO HadoopRDD: Input split: file:/D:/UNIPD/Big data/BD1819/text-sample.txt:20397128+20397128
19/04/11 16:52:19 INFO HadoopRDD: Input split: file:/D:/UNIPD/Big data/BD1819/text-sample.txt:0+20397128
19/04/11 16:52:19 INFO MemoryStore: Block rdd_1_0 stored as values in memory (estimated size 37.5 MB, free 1946.7 MB)
19/04/11 16:52:19 INFO BlockManagerInfo: Added rdd_1_0 in memory on 192.168.56.1:52183 (size: 37.5 MB, free: 1947.0 MB)
19/04/11 16:52:19 INFO MemoryStore: Block rdd_1_1 stored as values in memory (estimated size 37.6 MB, free 1909.1 MB)
19/04/11 16:52:19 INFO BlockManagerInfo: Added rdd_1_1 in memory on 192.168.56.1:52183 (size: 37.6 MB, free: 1909.4 MB)
19/04/11 16:52:19 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1572 bytes result sent to driver
19/04/11 16:52:19 INFO Executor: Finished task 1.0 in stage 0.0 (TID 1). 1572 bytes result sent to driver
19/04/11 16:52:19 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 405 ms on localhost (executor driver) (1/2)
19/04/11 16:52:19 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 415 ms on localhost (executor driver) (2/2)
19/04/11 16:52:19 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
19/04/11 16:52:19 INFO DAGScheduler: ResultStage 0 (count at G14HM2.java:31) finished in 0.426 s
19/04/11 16:52:19 INFO DAGScheduler: Job 0 finished: count at G14HM2.java:31, took 0.516603 s
19/04/11 16:52:20 INFO SparkContext: Starting job: count at G14HM2.java:83
19/04/11 16:52:20 INFO DAGScheduler: Registering RDD 6 (flatMapToPair at G14HM2.java:62)
19/04/11 16:52:20 INFO DAGScheduler: Got job 1 (count at G14HM2.java:83) with 2 output partitions
19/04/11 16:52:20 INFO DAGScheduler: Final stage: ResultStage 2 (count at G14HM2.java:83)
19/04/11 16:52:20 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 1)
19/04/11 16:52:20 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 1)
19/04/11 16:52:20 INFO DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[6] at flatMapToPair at G14HM2.java:62), which has no missing parents
19/04/11 16:52:20 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 5.2 KB, free 1909.1 MB)
19/04/11 16:52:20 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.9 KB, free 1909.1 MB)
19/04/11 16:52:20 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.56.1:52183 (size: 2.9 KB, free: 1909.4 MB)
19/04/11 16:52:20 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006
19/04/11 16:52:20 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[6] at flatMapToPair at G14HM2.java:62) (first 15 tasks are for partitions Vector(0, 1))
19/04/11 16:52:20 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks
19/04/11 16:52:20 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, localhost, executor driver, partition 0, PROCESS_LOCAL, 4854 bytes)
19/04/11 16:52:20 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 3, localhost, executor driver, partition 1, PROCESS_LOCAL, 4854 bytes)
19/04/11 16:52:20 INFO Executor: Running task 1.0 in stage 1.0 (TID 3)
19/04/11 16:52:20 INFO Executor: Running task 0.0 in stage 1.0 (TID 2)
19/04/11 16:52:20 INFO BlockManager: Found block rdd_1_0 locally
19/04/11 16:52:20 INFO BlockManager: Found block rdd_1_1 locally
19/04/11 16:52:20 INFO Executor: Finished task 1.0 in stage 1.0 (TID 3). 1154 bytes result sent to driver
19/04/11 16:52:20 INFO Executor: Finished task 0.0 in stage 1.0 (TID 2). 1197 bytes result sent to driver
19/04/11 16:52:20 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 708 ms on localhost (executor driver) (1/2)
19/04/11 16:52:20 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 3) in 706 ms on localhost (executor driver) (2/2)
19/04/11 16:52:20 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
19/04/11 16:52:20 INFO DAGScheduler: ShuffleMapStage 1 (flatMapToPair at G14HM2.java:62) finished in 0.710 s
19/04/11 16:52:20 INFO DAGScheduler: looking for newly runnable stages
19/04/11 16:52:20 INFO DAGScheduler: running: Set()
19/04/11 16:52:20 INFO DAGScheduler: waiting: Set(ResultStage 2)
19/04/11 16:52:20 INFO DAGScheduler: failed: Set()
19/04/11 16:52:20 INFO DAGScheduler: Submitting ResultStage 2 (ShuffledRDD[7] at reduceByKey at G14HM2.java:80), which has no missing parents
19/04/11 16:52:20 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 3.5 KB, free 1909.1 MB)
19/04/11 16:52:20 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 2.0 KB, free 1909.1 MB)
19/04/11 16:52:20 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 192.168.56.1:52183 (size: 2.0 KB, free: 1909.4 MB)
19/04/11 16:52:20 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1006
19/04/11 16:52:20 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 2 (ShuffledRDD[7] at reduceByKey at G14HM2.java:80) (first 15 tasks are for partitions Vector(0, 1))
19/04/11 16:52:20 INFO TaskSchedulerImpl: Adding task set 2.0 with 2 tasks
19/04/11 16:52:20 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 4, localhost, executor driver, partition 0, ANY, 4621 bytes)
19/04/11 16:52:20 INFO TaskSetManager: Starting task 1.0 in stage 2.0 (TID 5, localhost, executor driver, partition 1, ANY, 4621 bytes)
19/04/11 16:52:20 INFO Executor: Running task 0.0 in stage 2.0 (TID 4)
19/04/11 16:52:20 INFO Executor: Running task 1.0 in stage 2.0 (TID 5)
19/04/11 16:52:20 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:20 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 3 ms
19/04/11 16:52:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 3 ms
19/04/11 16:52:21 INFO MemoryStore: Block rdd_7_1 stored as values in memory (estimated size 8.4 MB, free 1895.1 MB)
19/04/11 16:52:21 INFO BlockManagerInfo: Added rdd_7_1 in memory on 192.168.56.1:52183 (size: 8.4 MB, free: 1900.9 MB)
19/04/11 16:52:21 INFO Executor: Finished task 1.0 in stage 2.0 (TID 5). 1830 bytes result sent to driver
19/04/11 16:52:21 INFO MemoryStore: Block rdd_7_0 stored as values in memory (estimated size 8.6 MB, free 1892.1 MB)
19/04/11 16:52:21 INFO BlockManagerInfo: Added rdd_7_0 in memory on 192.168.56.1:52183 (size: 8.6 MB, free: 1892.3 MB)
19/04/11 16:52:21 INFO TaskSetManager: Finished task 1.0 in stage 2.0 (TID 5) in 365 ms on localhost (executor driver) (1/2)
19/04/11 16:52:21 INFO Executor: Finished task 0.0 in stage 2.0 (TID 4). 1873 bytes result sent to driver
19/04/11 16:52:21 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 4) in 368 ms on localhost (executor driver) (2/2)
19/04/11 16:52:21 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
19/04/11 16:52:21 INFO DAGScheduler: ResultStage 2 (count at G14HM2.java:83) finished in 0.369 s
19/04/11 16:52:21 INFO DAGScheduler: Job 1 finished: count at G14HM2.java:83, took 1.108714 s
Execution time of Improved Word Count 1: 1125ms
19/04/11 16:52:21 INFO SparkContext: Starting job: count at G14HM2.java:147
19/04/11 16:52:21 INFO DAGScheduler: Registering RDD 9 (groupBy at G14HM2.java:110)
19/04/11 16:52:21 INFO DAGScheduler: Registering RDD 12 (flatMapToPair at G14HM2.java:119)
19/04/11 16:52:21 INFO DAGScheduler: Got job 2 (count at G14HM2.java:147) with 2 output partitions
19/04/11 16:52:21 INFO DAGScheduler: Final stage: ResultStage 5 (count at G14HM2.java:147)
19/04/11 16:52:21 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 4)
19/04/11 16:52:21 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 4)
19/04/11 16:52:21 INFO DAGScheduler: Submitting ShuffleMapStage 3 (MapPartitionsRDD[9] at groupBy at G14HM2.java:110), which has no missing parents
19/04/11 16:52:21 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 6.5 KB, free 1892.1 MB)
19/04/11 16:52:21 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 3.5 KB, free 1892.1 MB)
19/04/11 16:52:21 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 192.168.56.1:52183 (size: 3.5 KB, free: 1892.3 MB)
19/04/11 16:52:21 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006
19/04/11 16:52:21 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 3 (MapPartitionsRDD[9] at groupBy at G14HM2.java:110) (first 15 tasks are for partitions Vector(0, 1))
19/04/11 16:52:21 INFO TaskSchedulerImpl: Adding task set 3.0 with 2 tasks
19/04/11 16:52:21 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 6, localhost, executor driver, partition 0, PROCESS_LOCAL, 4854 bytes)
19/04/11 16:52:21 INFO TaskSetManager: Starting task 1.0 in stage 3.0 (TID 7, localhost, executor driver, partition 1, PROCESS_LOCAL, 4854 bytes)
19/04/11 16:52:21 INFO Executor: Running task 0.0 in stage 3.0 (TID 6)
19/04/11 16:52:21 INFO Executor: Running task 1.0 in stage 3.0 (TID 7)
19/04/11 16:52:21 INFO BlockManager: Found block rdd_1_1 locally
19/04/11 16:52:21 INFO BlockManager: Found block rdd_1_0 locally
19/04/11 16:52:21 INFO Executor: Finished task 1.0 in stage 3.0 (TID 7). 982 bytes result sent to driver
19/04/11 16:52:21 INFO TaskSetManager: Finished task 1.0 in stage 3.0 (TID 7) in 826 ms on localhost (executor driver) (1/2)
19/04/11 16:52:21 INFO Executor: Finished task 0.0 in stage 3.0 (TID 6). 982 bytes result sent to driver
19/04/11 16:52:21 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 6) in 838 ms on localhost (executor driver) (2/2)
19/04/11 16:52:21 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool
19/04/11 16:52:21 INFO DAGScheduler: ShuffleMapStage 3 (groupBy at G14HM2.java:110) finished in 0.838 s
19/04/11 16:52:21 INFO DAGScheduler: looking for newly runnable stages
19/04/11 16:52:21 INFO DAGScheduler: running: Set()
19/04/11 16:52:21 INFO DAGScheduler: waiting: Set(ResultStage 5, ShuffleMapStage 4)
19/04/11 16:52:21 INFO DAGScheduler: failed: Set()
19/04/11 16:52:21 INFO DAGScheduler: Submitting ShuffleMapStage 4 (MapPartitionsRDD[12] at flatMapToPair at G14HM2.java:119), which has no missing parents
19/04/11 16:52:21 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 8.2 KB, free 1892.1 MB)
19/04/11 16:52:21 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 4.1 KB, free 1892.1 MB)
19/04/11 16:52:21 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on 192.168.56.1:52183 (size: 4.1 KB, free: 1892.3 MB)
19/04/11 16:52:21 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1006
19/04/11 16:52:21 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 4 (MapPartitionsRDD[12] at flatMapToPair at G14HM2.java:119) (first 15 tasks are for partitions Vector(0, 1))
19/04/11 16:52:21 INFO TaskSchedulerImpl: Adding task set 4.0 with 2 tasks
19/04/11 16:52:21 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 8, localhost, executor driver, partition 0, ANY, 4610 bytes)
19/04/11 16:52:21 INFO TaskSetManager: Starting task 1.0 in stage 4.0 (TID 9, localhost, executor driver, partition 1, ANY, 4610 bytes)
19/04/11 16:52:21 INFO Executor: Running task 1.0 in stage 4.0 (TID 9)
19/04/11 16:52:21 INFO Executor: Running task 0.0 in stage 4.0 (TID 8)
19/04/11 16:52:22 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:22 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:22 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:22 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:22 INFO BlockManagerInfo: Removed broadcast_4_piece0 on 192.168.56.1:52183 in memory (size: 3.5 KB, free: 1892.3 MB)
19/04/11 16:52:22 INFO BlockManagerInfo: Removed broadcast_3_piece0 on 192.168.56.1:52183 in memory (size: 2.0 KB, free: 1892.3 MB)
19/04/11 16:52:22 INFO BlockManagerInfo: Removed broadcast_1_piece0 on 192.168.56.1:52183 in memory (size: 1884.0 B, free: 1892.3 MB)
19/04/11 16:52:22 INFO BlockManagerInfo: Removed broadcast_2_piece0 on 192.168.56.1:52183 in memory (size: 2.9 KB, free: 1892.3 MB)
19/04/11 16:52:23 INFO Executor: Finished task 1.0 in stage 4.0 (TID 9). 1283 bytes result sent to driver
19/04/11 16:52:23 INFO TaskSetManager: Finished task 1.0 in stage 4.0 (TID 9) in 1176 ms on localhost (executor driver) (1/2)
19/04/11 16:52:23 INFO Executor: Finished task 0.0 in stage 4.0 (TID 8). 1283 bytes result sent to driver
19/04/11 16:52:23 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 8) in 1252 ms on localhost (executor driver) (2/2)
19/04/11 16:52:23 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool
19/04/11 16:52:23 INFO DAGScheduler: ShuffleMapStage 4 (flatMapToPair at G14HM2.java:119) finished in 1.253 s
19/04/11 16:52:23 INFO DAGScheduler: looking for newly runnable stages
19/04/11 16:52:23 INFO DAGScheduler: running: Set()
19/04/11 16:52:23 INFO DAGScheduler: waiting: Set(ResultStage 5)
19/04/11 16:52:23 INFO DAGScheduler: failed: Set()
19/04/11 16:52:23 INFO DAGScheduler: Submitting ResultStage 5 (ShuffledRDD[13] at reduceByKey at G14HM2.java:144), which has no missing parents
19/04/11 16:52:23 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 3.5 KB, free 1892.1 MB)
19/04/11 16:52:23 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 2.1 KB, free 1892.1 MB)
19/04/11 16:52:23 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on 192.168.56.1:52183 (size: 2.1 KB, free: 1892.3 MB)
19/04/11 16:52:23 INFO SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1006
19/04/11 16:52:23 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 5 (ShuffledRDD[13] at reduceByKey at G14HM2.java:144) (first 15 tasks are for partitions Vector(0, 1))
19/04/11 16:52:23 INFO TaskSchedulerImpl: Adding task set 5.0 with 2 tasks
19/04/11 16:52:23 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 10, localhost, executor driver, partition 0, ANY, 4621 bytes)
19/04/11 16:52:23 INFO TaskSetManager: Starting task 1.0 in stage 5.0 (TID 11, localhost, executor driver, partition 1, ANY, 4621 bytes)
19/04/11 16:52:23 INFO Executor: Running task 0.0 in stage 5.0 (TID 10)
19/04/11 16:52:23 INFO Executor: Running task 1.0 in stage 5.0 (TID 11)
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:23 INFO MemoryStore: Block rdd_13_1 stored as values in memory (estimated size 8.6 MB, free 1883.5 MB)
19/04/11 16:52:23 INFO BlockManagerInfo: Added rdd_13_1 in memory on 192.168.56.1:52183 (size: 8.6 MB, free: 1883.7 MB)
19/04/11 16:52:23 INFO MemoryStore: Block rdd_13_0 stored as values in memory (estimated size 8.7 MB, free 1874.8 MB)
19/04/11 16:52:23 INFO BlockManagerInfo: Added rdd_13_0 in memory on 192.168.56.1:52183 (size: 8.7 MB, free: 1875.0 MB)
19/04/11 16:52:23 INFO Executor: Finished task 1.0 in stage 5.0 (TID 11). 1787 bytes result sent to driver
19/04/11 16:52:23 INFO Executor: Finished task 0.0 in stage 5.0 (TID 10). 1787 bytes result sent to driver
19/04/11 16:52:23 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 10) in 181 ms on localhost (executor driver) (1/2)
19/04/11 16:52:23 INFO TaskSetManager: Finished task 1.0 in stage 5.0 (TID 11) in 181 ms on localhost (executor driver) (2/2)
19/04/11 16:52:23 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool
19/04/11 16:52:23 INFO DAGScheduler: ResultStage 5 (count at G14HM2.java:147) finished in 0.181 s
19/04/11 16:52:23 INFO DAGScheduler: Job 2 finished: count at G14HM2.java:147, took 2.291416 s
Execution time of Improved Word Count 2 with random keys: 2317ms
19/04/11 16:52:23 INFO SparkContext: Starting job: count at G14HM2.java:199
19/04/11 16:52:23 INFO DAGScheduler: Registering RDD 2 (repartition at G14HM2.java:29)
19/04/11 16:52:23 INFO DAGScheduler: Registering RDD 15 (mapPartitionsToPair at G14HM2.java:175)
19/04/11 16:52:23 INFO DAGScheduler: Got job 3 (count at G14HM2.java:199) with 18 output partitions
19/04/11 16:52:23 INFO DAGScheduler: Final stage: ResultStage 8 (count at G14HM2.java:199)
19/04/11 16:52:23 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 7)
19/04/11 16:52:23 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 7)
19/04/11 16:52:23 INFO DAGScheduler: Submitting ShuffleMapStage 6 (MapPartitionsRDD[2] at repartition at G14HM2.java:29), which has no missing parents
19/04/11 16:52:23 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 4.6 KB, free 1874.8 MB)
19/04/11 16:52:23 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 2.7 KB, free 1874.8 MB)
19/04/11 16:52:23 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on 192.168.56.1:52183 (size: 2.7 KB, free: 1875.0 MB)
19/04/11 16:52:23 INFO SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1006
19/04/11 16:52:23 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 6 (MapPartitionsRDD[2] at repartition at G14HM2.java:29) (first 15 tasks are for partitions Vector(0, 1))
19/04/11 16:52:23 INFO TaskSchedulerImpl: Adding task set 6.0 with 2 tasks
19/04/11 16:52:23 INFO TaskSetManager: Starting task 0.0 in stage 6.0 (TID 12, localhost, executor driver, partition 0, PROCESS_LOCAL, 4854 bytes)
19/04/11 16:52:23 INFO TaskSetManager: Starting task 1.0 in stage 6.0 (TID 13, localhost, executor driver, partition 1, PROCESS_LOCAL, 4854 bytes)
19/04/11 16:52:23 INFO Executor: Running task 1.0 in stage 6.0 (TID 13)
19/04/11 16:52:23 INFO Executor: Running task 0.0 in stage 6.0 (TID 12)
19/04/11 16:52:23 INFO BlockManager: Found block rdd_1_1 locally
19/04/11 16:52:23 INFO BlockManager: Found block rdd_1_0 locally
19/04/11 16:52:23 INFO Executor: Finished task 0.0 in stage 6.0 (TID 12). 955 bytes result sent to driver
19/04/11 16:52:23 INFO TaskSetManager: Finished task 0.0 in stage 6.0 (TID 12) in 333 ms on localhost (executor driver) (1/2)
19/04/11 16:52:23 INFO Executor: Finished task 1.0 in stage 6.0 (TID 13). 955 bytes result sent to driver
19/04/11 16:52:23 INFO TaskSetManager: Finished task 1.0 in stage 6.0 (TID 13) in 335 ms on localhost (executor driver) (2/2)
19/04/11 16:52:23 INFO TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool
19/04/11 16:52:23 INFO DAGScheduler: ShuffleMapStage 6 (repartition at G14HM2.java:29) finished in 0.335 s
19/04/11 16:52:23 INFO DAGScheduler: looking for newly runnable stages
19/04/11 16:52:23 INFO DAGScheduler: running: Set()
19/04/11 16:52:23 INFO DAGScheduler: waiting: Set(ShuffleMapStage 7, ResultStage 8)
19/04/11 16:52:23 INFO DAGScheduler: failed: Set()
19/04/11 16:52:23 INFO DAGScheduler: Submitting ShuffleMapStage 7 (MapPartitionsRDD[15] at mapPartitionsToPair at G14HM2.java:175), which has no missing parents
19/04/11 16:52:23 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 5.5 KB, free 1874.8 MB)
19/04/11 16:52:23 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 3.0 KB, free 1874.8 MB)
19/04/11 16:52:23 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on 192.168.56.1:52183 (size: 3.0 KB, free: 1875.0 MB)
19/04/11 16:52:23 INFO SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1006
19/04/11 16:52:23 INFO DAGScheduler: Submitting 18 missing tasks from ShuffleMapStage 7 (MapPartitionsRDD[15] at mapPartitionsToPair at G14HM2.java:175) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
19/04/11 16:52:23 INFO TaskSchedulerImpl: Adding task set 7.0 with 18 tasks
19/04/11 16:52:23 INFO TaskSetManager: Starting task 0.0 in stage 7.0 (TID 14, localhost, executor driver, partition 0, ANY, 4886 bytes)
19/04/11 16:52:23 INFO TaskSetManager: Starting task 1.0 in stage 7.0 (TID 15, localhost, executor driver, partition 1, ANY, 4886 bytes)
19/04/11 16:52:23 INFO TaskSetManager: Starting task 2.0 in stage 7.0 (TID 16, localhost, executor driver, partition 2, ANY, 4886 bytes)
19/04/11 16:52:23 INFO TaskSetManager: Starting task 3.0 in stage 7.0 (TID 17, localhost, executor driver, partition 3, ANY, 4886 bytes)
19/04/11 16:52:23 INFO TaskSetManager: Starting task 4.0 in stage 7.0 (TID 18, localhost, executor driver, partition 4, ANY, 4886 bytes)
19/04/11 16:52:23 INFO TaskSetManager: Starting task 5.0 in stage 7.0 (TID 19, localhost, executor driver, partition 5, ANY, 4886 bytes)
19/04/11 16:52:23 INFO TaskSetManager: Starting task 6.0 in stage 7.0 (TID 20, localhost, executor driver, partition 6, ANY, 4886 bytes)
19/04/11 16:52:23 INFO TaskSetManager: Starting task 7.0 in stage 7.0 (TID 21, localhost, executor driver, partition 7, ANY, 4886 bytes)
19/04/11 16:52:23 INFO TaskSetManager: Starting task 8.0 in stage 7.0 (TID 22, localhost, executor driver, partition 8, ANY, 4886 bytes)
19/04/11 16:52:23 INFO TaskSetManager: Starting task 9.0 in stage 7.0 (TID 23, localhost, executor driver, partition 9, ANY, 4886 bytes)
19/04/11 16:52:23 INFO TaskSetManager: Starting task 10.0 in stage 7.0 (TID 24, localhost, executor driver, partition 10, ANY, 4886 bytes)
19/04/11 16:52:23 INFO TaskSetManager: Starting task 11.0 in stage 7.0 (TID 25, localhost, executor driver, partition 11, ANY, 4886 bytes)
19/04/11 16:52:23 INFO Executor: Running task 0.0 in stage 7.0 (TID 14)
19/04/11 16:52:23 INFO Executor: Running task 1.0 in stage 7.0 (TID 15)
19/04/11 16:52:23 INFO Executor: Running task 2.0 in stage 7.0 (TID 16)
19/04/11 16:52:23 INFO Executor: Running task 3.0 in stage 7.0 (TID 17)
19/04/11 16:52:23 INFO Executor: Running task 5.0 in stage 7.0 (TID 19)
19/04/11 16:52:23 INFO Executor: Running task 4.0 in stage 7.0 (TID 18)
19/04/11 16:52:23 INFO Executor: Running task 6.0 in stage 7.0 (TID 20)
19/04/11 16:52:23 INFO Executor: Running task 8.0 in stage 7.0 (TID 22)
19/04/11 16:52:23 INFO Executor: Running task 7.0 in stage 7.0 (TID 21)
19/04/11 16:52:23 INFO Executor: Running task 10.0 in stage 7.0 (TID 24)
19/04/11 16:52:23 INFO Executor: Running task 9.0 in stage 7.0 (TID 23)
19/04/11 16:52:23 INFO Executor: Running task 11.0 in stage 7.0 (TID 25)
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:24 INFO BlockManagerInfo: Removed broadcast_7_piece0 on 192.168.56.1:52183 in memory (size: 2.7 KB, free: 1875.0 MB)
19/04/11 16:52:24 INFO Executor: Finished task 7.0 in stage 7.0 (TID 21). 1299 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Starting task 12.0 in stage 7.0 (TID 26, localhost, executor driver, partition 12, ANY, 4886 bytes)
19/04/11 16:52:24 INFO Executor: Running task 12.0 in stage 7.0 (TID 26)
19/04/11 16:52:24 INFO TaskSetManager: Finished task 7.0 in stage 7.0 (TID 21) in 402 ms on localhost (executor driver) (1/18)
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:24 INFO Executor: Finished task 8.0 in stage 7.0 (TID 22). 1342 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Starting task 13.0 in stage 7.0 (TID 27, localhost, executor driver, partition 13, ANY, 4886 bytes)
19/04/11 16:52:24 INFO TaskSetManager: Finished task 8.0 in stage 7.0 (TID 22) in 433 ms on localhost (executor driver) (2/18)
19/04/11 16:52:24 INFO Executor: Finished task 9.0 in stage 7.0 (TID 23). 1299 bytes result sent to driver
19/04/11 16:52:24 INFO Executor: Running task 13.0 in stage 7.0 (TID 27)
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:24 INFO TaskSetManager: Starting task 14.0 in stage 7.0 (TID 28, localhost, executor driver, partition 14, ANY, 4886 bytes)
19/04/11 16:52:24 INFO Executor: Running task 14.0 in stage 7.0 (TID 28)
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:24 INFO TaskSetManager: Finished task 9.0 in stage 7.0 (TID 23) in 441 ms on localhost (executor driver) (3/18)
19/04/11 16:52:24 INFO Executor: Finished task 6.0 in stage 7.0 (TID 20). 1299 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Starting task 15.0 in stage 7.0 (TID 29, localhost, executor driver, partition 15, ANY, 4886 bytes)
19/04/11 16:52:24 INFO TaskSetManager: Finished task 6.0 in stage 7.0 (TID 20) in 462 ms on localhost (executor driver) (4/18)
19/04/11 16:52:24 INFO Executor: Running task 15.0 in stage 7.0 (TID 29)
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
19/04/11 16:52:24 INFO Executor: Finished task 2.0 in stage 7.0 (TID 16). 1299 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Starting task 16.0 in stage 7.0 (TID 30, localhost, executor driver, partition 16, ANY, 4886 bytes)
19/04/11 16:52:24 INFO Executor: Finished task 5.0 in stage 7.0 (TID 19). 1299 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Starting task 17.0 in stage 7.0 (TID 31, localhost, executor driver, partition 17, ANY, 4886 bytes)
19/04/11 16:52:24 INFO Executor: Running task 16.0 in stage 7.0 (TID 30)
19/04/11 16:52:24 INFO TaskSetManager: Finished task 2.0 in stage 7.0 (TID 16) in 485 ms on localhost (executor driver) (5/18)
19/04/11 16:52:24 INFO Executor: Finished task 3.0 in stage 7.0 (TID 17). 1299 bytes result sent to driver
19/04/11 16:52:24 INFO Executor: Running task 17.0 in stage 7.0 (TID 31)
19/04/11 16:52:24 INFO TaskSetManager: Finished task 5.0 in stage 7.0 (TID 19) in 485 ms on localhost (executor driver) (6/18)
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 2 non-empty blocks out of 2 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:24 INFO TaskSetManager: Finished task 3.0 in stage 7.0 (TID 17) in 512 ms on localhost (executor driver) (7/18)
19/04/11 16:52:24 INFO Executor: Finished task 1.0 in stage 7.0 (TID 15). 1299 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Finished task 1.0 in stage 7.0 (TID 15) in 517 ms on localhost (executor driver) (8/18)
19/04/11 16:52:24 INFO Executor: Finished task 4.0 in stage 7.0 (TID 18). 1299 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Finished task 4.0 in stage 7.0 (TID 18) in 521 ms on localhost (executor driver) (9/18)
19/04/11 16:52:24 INFO Executor: Finished task 10.0 in stage 7.0 (TID 24). 1299 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Finished task 10.0 in stage 7.0 (TID 24) in 523 ms on localhost (executor driver) (10/18)
19/04/11 16:52:24 INFO Executor: Finished task 11.0 in stage 7.0 (TID 25). 1299 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Finished task 11.0 in stage 7.0 (TID 25) in 525 ms on localhost (executor driver) (11/18)
19/04/11 16:52:24 INFO Executor: Finished task 0.0 in stage 7.0 (TID 14). 1299 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Finished task 0.0 in stage 7.0 (TID 14) in 535 ms on localhost (executor driver) (12/18)
19/04/11 16:52:24 INFO Executor: Finished task 12.0 in stage 7.0 (TID 26). 1256 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Finished task 12.0 in stage 7.0 (TID 26) in 208 ms on localhost (executor driver) (13/18)
19/04/11 16:52:24 INFO Executor: Finished task 14.0 in stage 7.0 (TID 28). 1256 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Finished task 14.0 in stage 7.0 (TID 28) in 185 ms on localhost (executor driver) (14/18)
19/04/11 16:52:24 INFO Executor: Finished task 13.0 in stage 7.0 (TID 27). 1299 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Finished task 13.0 in stage 7.0 (TID 27) in 200 ms on localhost (executor driver) (15/18)
19/04/11 16:52:24 INFO Executor: Finished task 15.0 in stage 7.0 (TID 29). 1256 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Finished task 15.0 in stage 7.0 (TID 29) in 201 ms on localhost (executor driver) (16/18)
19/04/11 16:52:24 INFO Executor: Finished task 17.0 in stage 7.0 (TID 31). 1256 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Finished task 17.0 in stage 7.0 (TID 31) in 186 ms on localhost (executor driver) (17/18)
19/04/11 16:52:24 INFO Executor: Finished task 16.0 in stage 7.0 (TID 30). 1256 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Finished task 16.0 in stage 7.0 (TID 30) in 196 ms on localhost (executor driver) (18/18)
19/04/11 16:52:24 INFO TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool
19/04/11 16:52:24 INFO DAGScheduler: ShuffleMapStage 7 (mapPartitionsToPair at G14HM2.java:175) finished in 0.682 s
19/04/11 16:52:24 INFO DAGScheduler: looking for newly runnable stages
19/04/11 16:52:24 INFO DAGScheduler: running: Set()
19/04/11 16:52:24 INFO DAGScheduler: waiting: Set(ResultStage 8)
19/04/11 16:52:24 INFO DAGScheduler: failed: Set()
19/04/11 16:52:24 INFO DAGScheduler: Submitting ResultStage 8 (ShuffledRDD[16] at reduceByKey at G14HM2.java:196), which has no missing parents
19/04/11 16:52:24 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 3.5 KB, free 1874.8 MB)
19/04/11 16:52:24 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 2.1 KB, free 1874.8 MB)
19/04/11 16:52:24 INFO BlockManagerInfo: Added broadcast_9_piece0 in memory on 192.168.56.1:52183 (size: 2.1 KB, free: 1875.0 MB)
19/04/11 16:52:24 INFO SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:1006
19/04/11 16:52:24 INFO DAGScheduler: Submitting 18 missing tasks from ResultStage 8 (ShuffledRDD[16] at reduceByKey at G14HM2.java:196) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
19/04/11 16:52:24 INFO TaskSchedulerImpl: Adding task set 8.0 with 18 tasks
19/04/11 16:52:24 INFO TaskSetManager: Starting task 0.0 in stage 8.0 (TID 32, localhost, executor driver, partition 0, ANY, 4621 bytes)
19/04/11 16:52:24 INFO TaskSetManager: Starting task 1.0 in stage 8.0 (TID 33, localhost, executor driver, partition 1, ANY, 4621 bytes)
19/04/11 16:52:24 INFO TaskSetManager: Starting task 2.0 in stage 8.0 (TID 34, localhost, executor driver, partition 2, ANY, 4621 bytes)
19/04/11 16:52:24 INFO TaskSetManager: Starting task 3.0 in stage 8.0 (TID 35, localhost, executor driver, partition 3, ANY, 4621 bytes)
19/04/11 16:52:24 INFO TaskSetManager: Starting task 4.0 in stage 8.0 (TID 36, localhost, executor driver, partition 4, ANY, 4621 bytes)
19/04/11 16:52:24 INFO TaskSetManager: Starting task 5.0 in stage 8.0 (TID 37, localhost, executor driver, partition 5, ANY, 4621 bytes)
19/04/11 16:52:24 INFO TaskSetManager: Starting task 6.0 in stage 8.0 (TID 38, localhost, executor driver, partition 6, ANY, 4621 bytes)
19/04/11 16:52:24 INFO TaskSetManager: Starting task 7.0 in stage 8.0 (TID 39, localhost, executor driver, partition 7, ANY, 4621 bytes)
19/04/11 16:52:24 INFO TaskSetManager: Starting task 8.0 in stage 8.0 (TID 40, localhost, executor driver, partition 8, ANY, 4621 bytes)
19/04/11 16:52:24 INFO TaskSetManager: Starting task 9.0 in stage 8.0 (TID 41, localhost, executor driver, partition 9, ANY, 4621 bytes)
19/04/11 16:52:24 INFO TaskSetManager: Starting task 10.0 in stage 8.0 (TID 42, localhost, executor driver, partition 10, ANY, 4621 bytes)
19/04/11 16:52:24 INFO TaskSetManager: Starting task 11.0 in stage 8.0 (TID 43, localhost, executor driver, partition 11, ANY, 4621 bytes)
19/04/11 16:52:24 INFO Executor: Running task 1.0 in stage 8.0 (TID 33)
19/04/11 16:52:24 INFO Executor: Running task 0.0 in stage 8.0 (TID 32)
19/04/11 16:52:24 INFO Executor: Running task 6.0 in stage 8.0 (TID 38)
19/04/11 16:52:24 INFO Executor: Running task 5.0 in stage 8.0 (TID 37)
19/04/11 16:52:24 INFO Executor: Running task 4.0 in stage 8.0 (TID 36)
19/04/11 16:52:24 INFO Executor: Running task 3.0 in stage 8.0 (TID 35)
19/04/11 16:52:24 INFO Executor: Running task 2.0 in stage 8.0 (TID 34)
19/04/11 16:52:24 INFO Executor: Running task 10.0 in stage 8.0 (TID 42)
19/04/11 16:52:24 INFO Executor: Running task 7.0 in stage 8.0 (TID 39)
19/04/11 16:52:24 INFO Executor: Running task 9.0 in stage 8.0 (TID 41)
19/04/11 16:52:24 INFO Executor: Running task 11.0 in stage 8.0 (TID 43)
19/04/11 16:52:24 INFO Executor: Running task 8.0 in stage 8.0 (TID 40)
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 18 non-empty blocks out of 18 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 18 non-empty blocks out of 18 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 18 non-empty blocks out of 18 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 18 non-empty blocks out of 18 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 18 non-empty blocks out of 18 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 18 non-empty blocks out of 18 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 18 non-empty blocks out of 18 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 18 non-empty blocks out of 18 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 18 non-empty blocks out of 18 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 18 non-empty blocks out of 18 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 18 non-empty blocks out of 18 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 18 non-empty blocks out of 18 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:24 INFO MemoryStore: Block rdd_16_5 stored as values in memory (estimated size 940.5 KB, free 1872.9 MB)
19/04/11 16:52:24 INFO MemoryStore: Block rdd_16_4 stored as values in memory (estimated size 957.4 KB, free 1872.9 MB)
19/04/11 16:52:24 INFO BlockManagerInfo: Added rdd_16_5 in memory on 192.168.56.1:52183 (size: 940.5 KB, free: 1874.1 MB)
19/04/11 16:52:24 INFO BlockManagerInfo: Added rdd_16_4 in memory on 192.168.56.1:52183 (size: 957.4 KB, free: 1873.2 MB)
19/04/11 16:52:24 INFO MemoryStore: Block rdd_16_1 stored as values in memory (estimated size 954.4 KB, free 1872.0 MB)
19/04/11 16:52:24 INFO BlockManagerInfo: Added rdd_16_1 in memory on 192.168.56.1:52183 (size: 954.4 KB, free: 1872.2 MB)
19/04/11 16:52:24 INFO Executor: Finished task 1.0 in stage 8.0 (TID 33). 1830 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Starting task 12.0 in stage 8.0 (TID 44, localhost, executor driver, partition 12, ANY, 4621 bytes)
19/04/11 16:52:24 INFO Executor: Running task 12.0 in stage 8.0 (TID 44)
19/04/11 16:52:24 INFO TaskSetManager: Finished task 1.0 in stage 8.0 (TID 33) in 128 ms on localhost (executor driver) (1/18)
19/04/11 16:52:24 INFO Executor: Finished task 4.0 in stage 8.0 (TID 36). 1830 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Starting task 13.0 in stage 8.0 (TID 45, localhost, executor driver, partition 13, ANY, 4621 bytes)
19/04/11 16:52:24 INFO Executor: Running task 13.0 in stage 8.0 (TID 45)
19/04/11 16:52:24 INFO TaskSetManager: Finished task 4.0 in stage 8.0 (TID 36) in 129 ms on localhost (executor driver) (2/18)
19/04/11 16:52:24 INFO MemoryStore: Block rdd_16_11 stored as values in memory (estimated size 945.1 KB, free 1870.1 MB)
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 18 non-empty blocks out of 18 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:24 INFO BlockManagerInfo: Added rdd_16_11 in memory on 192.168.56.1:52183 (size: 945.1 KB, free: 1871.3 MB)
19/04/11 16:52:24 INFO MemoryStore: Block rdd_16_9 stored as values in memory (estimated size 957.8 KB, free 1869.2 MB)
19/04/11 16:52:24 INFO BlockManagerInfo: Added rdd_16_9 in memory on 192.168.56.1:52183 (size: 957.8 KB, free: 1870.4 MB)
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 18 non-empty blocks out of 18 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:24 INFO MemoryStore: Block rdd_16_0 stored as values in memory (estimated size 957.0 KB, free 1868.3 MB)
19/04/11 16:52:24 INFO Executor: Finished task 11.0 in stage 8.0 (TID 43). 1830 bytes result sent to driver
19/04/11 16:52:24 INFO BlockManagerInfo: Added rdd_16_0 in memory on 192.168.56.1:52183 (size: 957.0 KB, free: 1869.4 MB)
19/04/11 16:52:24 INFO TaskSetManager: Starting task 14.0 in stage 8.0 (TID 46, localhost, executor driver, partition 14, ANY, 4621 bytes)
19/04/11 16:52:24 INFO Executor: Finished task 5.0 in stage 8.0 (TID 37). 1830 bytes result sent to driver
19/04/11 16:52:24 INFO Executor: Finished task 9.0 in stage 8.0 (TID 41). 1830 bytes result sent to driver
19/04/11 16:52:24 INFO Executor: Running task 14.0 in stage 8.0 (TID 46)
19/04/11 16:52:24 INFO TaskSetManager: Starting task 15.0 in stage 8.0 (TID 47, localhost, executor driver, partition 15, ANY, 4621 bytes)
19/04/11 16:52:24 INFO Executor: Running task 15.0 in stage 8.0 (TID 47)
19/04/11 16:52:24 INFO Executor: Finished task 0.0 in stage 8.0 (TID 32). 1830 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Starting task 16.0 in stage 8.0 (TID 48, localhost, executor driver, partition 16, ANY, 4621 bytes)
19/04/11 16:52:24 INFO Executor: Running task 16.0 in stage 8.0 (TID 48)
19/04/11 16:52:24 INFO TaskSetManager: Starting task 17.0 in stage 8.0 (TID 49, localhost, executor driver, partition 17, ANY, 4621 bytes)
19/04/11 16:52:24 INFO Executor: Running task 17.0 in stage 8.0 (TID 49)
19/04/11 16:52:24 INFO TaskSetManager: Finished task 5.0 in stage 8.0 (TID 37) in 133 ms on localhost (executor driver) (3/18)
19/04/11 16:52:24 INFO TaskSetManager: Finished task 0.0 in stage 8.0 (TID 32) in 134 ms on localhost (executor driver) (4/18)
19/04/11 16:52:24 INFO TaskSetManager: Finished task 11.0 in stage 8.0 (TID 43) in 133 ms on localhost (executor driver) (5/18)
19/04/11 16:52:24 INFO TaskSetManager: Finished task 9.0 in stage 8.0 (TID 41) in 133 ms on localhost (executor driver) (6/18)
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 18 non-empty blocks out of 18 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 18 non-empty blocks out of 18 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 18 non-empty blocks out of 18 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Getting 18 non-empty blocks out of 18 blocks
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
19/04/11 16:52:24 INFO MemoryStore: Block rdd_16_10 stored as values in memory (estimated size 988.8 KB, free 1867.3 MB)
19/04/11 16:52:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
19/04/11 16:52:24 INFO MemoryStore: Block rdd_16_3 stored as values in memory (estimated size 928.9 KB, free 1866.4 MB)
19/04/11 16:52:24 INFO BlockManagerInfo: Added rdd_16_10 in memory on 192.168.56.1:52183 (size: 988.8 KB, free: 1868.5 MB)
19/04/11 16:52:24 INFO BlockManagerInfo: Added rdd_16_3 in memory on 192.168.56.1:52183 (size: 928.9 KB, free: 1867.6 MB)
19/04/11 16:52:24 INFO MemoryStore: Block rdd_16_7 stored as values in memory (estimated size 966.9 KB, free 1865.5 MB)
19/04/11 16:52:24 INFO BlockManagerInfo: Added rdd_16_7 in memory on 192.168.56.1:52183 (size: 966.9 KB, free: 1866.6 MB)
19/04/11 16:52:24 INFO Executor: Finished task 10.0 in stage 8.0 (TID 42). 1830 bytes result sent to driver
19/04/11 16:52:24 INFO Executor: Finished task 7.0 in stage 8.0 (TID 39). 1830 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Finished task 10.0 in stage 8.0 (TID 42) in 144 ms on localhost (executor driver) (7/18)
19/04/11 16:52:24 INFO TaskSetManager: Finished task 7.0 in stage 8.0 (TID 39) in 144 ms on localhost (executor driver) (8/18)
19/04/11 16:52:24 INFO Executor: Finished task 3.0 in stage 8.0 (TID 35). 1830 bytes result sent to driver
19/04/11 16:52:24 INFO MemoryStore: Block rdd_16_2 stored as values in memory (estimated size 945.9 KB, free 1870.1 MB)
19/04/11 16:52:24 INFO TaskSetManager: Finished task 3.0 in stage 8.0 (TID 35) in 147 ms on localhost (executor driver) (9/18)
19/04/11 16:52:24 INFO BlockManagerInfo: Added rdd_16_2 in memory on 192.168.56.1:52183 (size: 945.9 KB, free: 1865.7 MB)
19/04/11 16:52:24 INFO Executor: Finished task 2.0 in stage 8.0 (TID 34). 1830 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Finished task 2.0 in stage 8.0 (TID 34) in 149 ms on localhost (executor driver) (10/18)
19/04/11 16:52:24 INFO MemoryStore: Block rdd_16_8 stored as values in memory (estimated size 969.1 KB, free 1864.5 MB)
19/04/11 16:52:24 INFO BlockManagerInfo: Added rdd_16_8 in memory on 192.168.56.1:52183 (size: 969.1 KB, free: 1864.8 MB)
19/04/11 16:52:24 INFO Executor: Finished task 8.0 in stage 8.0 (TID 40). 1830 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Finished task 8.0 in stage 8.0 (TID 40) in 158 ms on localhost (executor driver) (11/18)
19/04/11 16:52:24 INFO MemoryStore: Block rdd_16_6 stored as values in memory (estimated size 948.0 KB, free 1863.6 MB)
19/04/11 16:52:24 INFO BlockManagerInfo: Added rdd_16_6 in memory on 192.168.56.1:52183 (size: 948.0 KB, free: 1863.8 MB)
19/04/11 16:52:24 INFO Executor: Finished task 6.0 in stage 8.0 (TID 38). 1830 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Finished task 6.0 in stage 8.0 (TID 38) in 165 ms on localhost (executor driver) (12/18)
19/04/11 16:52:24 INFO MemoryStore: Block rdd_16_13 stored as values in memory (estimated size 956.1 KB, free 1862.7 MB)
19/04/11 16:52:24 INFO BlockManagerInfo: Added rdd_16_13 in memory on 192.168.56.1:52183 (size: 956.1 KB, free: 1862.9 MB)
19/04/11 16:52:24 INFO Executor: Finished task 13.0 in stage 8.0 (TID 45). 1744 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Finished task 13.0 in stage 8.0 (TID 45) in 69 ms on localhost (executor driver) (13/18)
19/04/11 16:52:24 INFO MemoryStore: Block rdd_16_15 stored as values in memory (estimated size 943.8 KB, free 1861.7 MB)
19/04/11 16:52:24 INFO MemoryStore: Block rdd_16_14 stored as values in memory (estimated size 918.1 KB, free 1860.8 MB)
19/04/11 16:52:24 INFO BlockManagerInfo: Added rdd_16_15 in memory on 192.168.56.1:52183 (size: 943.8 KB, free: 1862.0 MB)
19/04/11 16:52:24 INFO BlockManagerInfo: Added rdd_16_14 in memory on 192.168.56.1:52183 (size: 918.1 KB, free: 1861.1 MB)
19/04/11 16:52:24 INFO Executor: Finished task 15.0 in stage 8.0 (TID 47). 1744 bytes result sent to driver
19/04/11 16:52:24 INFO Executor: Finished task 14.0 in stage 8.0 (TID 46). 1744 bytes result sent to driver
19/04/11 16:52:24 INFO MemoryStore: Block rdd_16_16 stored as values in memory (estimated size 962.5 KB, free 1859.9 MB)
19/04/11 16:52:24 INFO TaskSetManager: Finished task 15.0 in stage 8.0 (TID 47) in 67 ms on localhost (executor driver) (14/18)
19/04/11 16:52:24 INFO BlockManagerInfo: Added rdd_16_16 in memory on 192.168.56.1:52183 (size: 962.5 KB, free: 1860.1 MB)
19/04/11 16:52:24 INFO TaskSetManager: Finished task 14.0 in stage 8.0 (TID 46) in 69 ms on localhost (executor driver) (15/18)
19/04/11 16:52:24 INFO Executor: Finished task 16.0 in stage 8.0 (TID 48). 1787 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Finished task 16.0 in stage 8.0 (TID 48) in 69 ms on localhost (executor driver) (16/18)
19/04/11 16:52:24 INFO MemoryStore: Block rdd_16_12 stored as values in memory (estimated size 926.5 KB, free 1859.0 MB)
19/04/11 16:52:24 INFO BlockManagerInfo: Added rdd_16_12 in memory on 192.168.56.1:52183 (size: 926.5 KB, free: 1859.2 MB)
19/04/11 16:52:24 INFO MemoryStore: Block rdd_16_17 stored as values in memory (estimated size 962.6 KB, free 1858.1 MB)
19/04/11 16:52:24 INFO Executor: Finished task 12.0 in stage 8.0 (TID 44). 1744 bytes result sent to driver
19/04/11 16:52:24 INFO BlockManagerInfo: Added rdd_16_17 in memory on 192.168.56.1:52183 (size: 962.6 KB, free: 1858.3 MB)
19/04/11 16:52:24 INFO TaskSetManager: Finished task 12.0 in stage 8.0 (TID 44) in 82 ms on localhost (executor driver) (17/18)
19/04/11 16:52:24 INFO Executor: Finished task 17.0 in stage 8.0 (TID 49). 1787 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Finished task 17.0 in stage 8.0 (TID 49) in 77 ms on localhost (executor driver) (18/18)
19/04/11 16:52:24 INFO TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool
19/04/11 16:52:24 INFO DAGScheduler: ResultStage 8 (count at G14HM2.java:199) finished in 0.210 s
19/04/11 16:52:24 INFO DAGScheduler: Job 3 finished: count at G14HM2.java:199, took 1.247318 s
Execution time of Improved Word Count 2 with Spark partition : 1262ms
19/04/11 16:52:24 INFO SparkContext: Starting job: reduce at G14HM2.java:210
19/04/11 16:52:24 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 157 bytes
19/04/11 16:52:24 INFO DAGScheduler: Got job 4 (reduce at G14HM2.java:210) with 2 output partitions
19/04/11 16:52:24 INFO DAGScheduler: Final stage: ResultStage 10 (reduce at G14HM2.java:210)
19/04/11 16:52:24 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 9)
19/04/11 16:52:24 INFO DAGScheduler: Missing parents: List()
19/04/11 16:52:24 INFO DAGScheduler: Submitting ResultStage 10 (MapPartitionsRDD[18] at map at G14HM2.java:210), which has no missing parents
19/04/11 16:52:24 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 4.5 KB, free 1858.0 MB)
19/04/11 16:52:24 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 2.5 KB, free 1858.0 MB)
19/04/11 16:52:24 INFO BlockManagerInfo: Added broadcast_10_piece0 in memory on 192.168.56.1:52183 (size: 2.5 KB, free: 1858.3 MB)
19/04/11 16:52:24 INFO SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:1006
19/04/11 16:52:24 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 10 (MapPartitionsRDD[18] at map at G14HM2.java:210) (first 15 tasks are for partitions Vector(0, 1))
19/04/11 16:52:24 INFO TaskSchedulerImpl: Adding task set 10.0 with 2 tasks
19/04/11 16:52:24 INFO TaskSetManager: Starting task 0.0 in stage 10.0 (TID 50, localhost, executor driver, partition 0, PROCESS_LOCAL, 4621 bytes)
19/04/11 16:52:24 INFO TaskSetManager: Starting task 1.0 in stage 10.0 (TID 51, localhost, executor driver, partition 1, PROCESS_LOCAL, 4621 bytes)
19/04/11 16:52:24 INFO Executor: Running task 1.0 in stage 10.0 (TID 51)
19/04/11 16:52:24 INFO Executor: Running task 0.0 in stage 10.0 (TID 50)
19/04/11 16:52:24 INFO BlockManager: Found block rdd_7_1 locally
19/04/11 16:52:24 INFO BlockManager: Found block rdd_7_0 locally
19/04/11 16:52:24 INFO Executor: Finished task 1.0 in stage 10.0 (TID 51). 867 bytes result sent to driver
19/04/11 16:52:24 INFO Executor: Finished task 0.0 in stage 10.0 (TID 50). 867 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Finished task 1.0 in stage 10.0 (TID 51) in 14 ms on localhost (executor driver) (1/2)
19/04/11 16:52:24 INFO TaskSetManager: Finished task 0.0 in stage 10.0 (TID 50) in 14 ms on localhost (executor driver) (2/2)
19/04/11 16:52:24 INFO TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool
19/04/11 16:52:24 INFO DAGScheduler: ResultStage 10 (reduce at G14HM2.java:210) finished in 0.014 s
19/04/11 16:52:24 INFO DAGScheduler: Job 4 finished: reduce at G14HM2.java:210, took 0.020254 s
19/04/11 16:52:24 INFO SparkContext: Starting job: count at G14HM2.java:213
19/04/11 16:52:24 INFO DAGScheduler: Got job 5 (count at G14HM2.java:213) with 2 output partitions
19/04/11 16:52:24 INFO DAGScheduler: Final stage: ResultStage 12 (count at G14HM2.java:213)
19/04/11 16:52:24 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 11)
19/04/11 16:52:24 INFO DAGScheduler: Missing parents: List()
19/04/11 16:52:24 INFO DAGScheduler: Submitting ResultStage 12 (MapPartitionsRDD[17] at keys at G14HM2.java:206), which has no missing parents
19/04/11 16:52:24 INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 3.9 KB, free 1858.0 MB)
19/04/11 16:52:24 INFO MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 2.3 KB, free 1858.0 MB)
19/04/11 16:52:24 INFO BlockManagerInfo: Added broadcast_11_piece0 in memory on 192.168.56.1:52183 (size: 2.3 KB, free: 1858.3 MB)
19/04/11 16:52:24 INFO SparkContext: Created broadcast 11 from broadcast at DAGScheduler.scala:1006
19/04/11 16:52:24 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 12 (MapPartitionsRDD[17] at keys at G14HM2.java:206) (first 15 tasks are for partitions Vector(0, 1))
19/04/11 16:52:24 INFO TaskSchedulerImpl: Adding task set 12.0 with 2 tasks
19/04/11 16:52:24 INFO TaskSetManager: Starting task 0.0 in stage 12.0 (TID 52, localhost, executor driver, partition 0, PROCESS_LOCAL, 4621 bytes)
19/04/11 16:52:24 INFO TaskSetManager: Starting task 1.0 in stage 12.0 (TID 53, localhost, executor driver, partition 1, PROCESS_LOCAL, 4621 bytes)
19/04/11 16:52:24 INFO Executor: Running task 0.0 in stage 12.0 (TID 52)
19/04/11 16:52:24 INFO Executor: Running task 1.0 in stage 12.0 (TID 53)
19/04/11 16:52:24 INFO BlockManager: Found block rdd_7_0 locally
19/04/11 16:52:24 INFO BlockManager: Found block rdd_7_1 locally
19/04/11 16:52:24 INFO Executor: Finished task 0.0 in stage 12.0 (TID 52). 703 bytes result sent to driver
19/04/11 16:52:24 INFO Executor: Finished task 1.0 in stage 12.0 (TID 53). 703 bytes result sent to driver
19/04/11 16:52:24 INFO TaskSetManager: Finished task 0.0 in stage 12.0 (TID 52) in 6 ms on localhost (executor driver) (1/2)
19/04/11 16:52:24 INFO TaskSetManager: Finished task 1.0 in stage 12.0 (TID 53) in 6 ms on localhost (executor driver) (2/2)
19/04/11 16:52:24 INFO TaskSchedulerImpl: Removed TaskSet 12.0, whose tasks have all completed, from pool
19/04/11 16:52:24 INFO DAGScheduler: ResultStage 12 (count at G14HM2.java:213) finished in 0.006 s
19/04/11 16:52:24 INFO DAGScheduler: Job 5 finished: count at G14HM2.java:213, took 0.009840 s
The average length of the distinct words is: 16.94939432303233
19/04/11 16:52:24 INFO SparkContext: Invoking stop() from shutdown hook
19/04/11 16:52:24 INFO SparkUI: Stopped Spark web UI at http://192.168.56.1:4040
19/04/11 16:52:24 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/04/11 16:52:24 INFO MemoryStore: MemoryStore cleared
19/04/11 16:52:24 INFO BlockManager: BlockManager stopped
19/04/11 16:52:24 INFO BlockManagerMaster: BlockManagerMaster stopped
19/04/11 16:52:24 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/04/11 16:52:24 INFO SparkContext: Successfully stopped SparkContext
19/04/11 16:52:24 INFO ShutdownHookManager: Shutdown hook called
19/04/11 16:52:24 INFO ShutdownHookManager: Deleting directory C:\Users\Raffaele_DNDM\AppData\Local\Temp\spark-96de02be-a730-4e7a-a529-dc72a70dfe70