Created
September 25, 2015 20:23
-
-
Save vmakhaev/b083feded49a38c69ca9 to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| scala> hc.sql("select * from price_equities_technicals where marketdate = 1443052800000 and sid = 21438 limit 1").show() | |
| 15/09/25 20:21:59 INFO parse.ParseDriver: Parsing command: select * from price_equities_technicals where marketdate = 1443052800000 and sid = 21438 limit 1 | |
| 15/09/25 20:21:59 INFO parse.ParseDriver: Parse Completed | |
| 15/09/25 20:21:59 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=price_equities_technicals | |
| 15/09/25 20:21:59 INFO HiveMetaStore.audit: ugi=hadoop ip=unknown-ip-addr cmd=get_table : db=default tbl=price_equities_technicals | |
| 15/09/25 20:21:59 INFO storage.BlockManager: Removing broadcast 10 | |
| 15/09/25 20:21:59 INFO storage.BlockManager: Removing block broadcast_10 | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: Block broadcast_10 of size 65056 dropped from memory (free 2221338296) | |
| 15/09/25 20:21:59 INFO storage.BlockManager: Removing block broadcast_10_piece0 | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: Block broadcast_10_piece0 of size 18019 dropped from memory (free 2221356315) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerInfo: Removed broadcast_10_piece0 on ip-10-171-120-202.ec2.internal:50304 in memory (size: 17.6 KB, free: 2.1 GB) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerMaster: Updated info of block broadcast_10_piece0 | |
| 15/09/25 20:21:59 INFO storage.BlockManagerInfo: Removed broadcast_10_piece0 on ip-10-69-77-8.ec2.internal:47777 in memory (size: 17.6 KB, free: 1069.8 MB) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerInfo: Removed broadcast_10_piece0 on ip-10-138-58-114.ec2.internal:44188 in memory (size: 17.6 KB, free: 1070.0 MB) | |
| 15/09/25 20:21:59 INFO spark.ContextCleaner: Cleaned broadcast 10 | |
| 15/09/25 20:21:59 INFO storage.BlockManager: Removing broadcast 9 | |
| 15/09/25 20:21:59 INFO storage.BlockManager: Removing block broadcast_9 | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: Block broadcast_9 of size 65056 dropped from memory (free 2221421371) | |
| 15/09/25 20:21:59 INFO storage.BlockManager: Removing block broadcast_9_piece0 | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: Block broadcast_9_piece0 of size 18019 dropped from memory (free 2221439390) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerInfo: Removed broadcast_9_piece0 on ip-10-171-120-202.ec2.internal:50304 in memory (size: 17.6 KB, free: 2.1 GB) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerMaster: Updated info of block broadcast_9_piece0 | |
| 15/09/25 20:21:59 INFO storage.BlockManagerInfo: Removed broadcast_9_piece0 on ip-10-69-77-8.ec2.internal:47777 in memory (size: 17.6 KB, free: 1069.9 MB) | |
| 15/09/25 20:21:59 INFO spark.ContextCleaner: Cleaned broadcast 9 | |
| 15/09/25 20:21:59 INFO storage.BlockManager: Removing broadcast 8 | |
| 15/09/25 20:21:59 INFO storage.BlockManager: Removing block broadcast_8 | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: Block broadcast_8 of size 432911 dropped from memory (free 2221872301) | |
| 15/09/25 20:21:59 INFO storage.BlockManager: Removing block broadcast_8_piece0 | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: Block broadcast_8_piece0 of size 39948 dropped from memory (free 2221912249) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerInfo: Removed broadcast_8_piece0 on ip-10-171-120-202.ec2.internal:50304 in memory (size: 39.0 KB, free: 2.1 GB) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerMaster: Updated info of block broadcast_8_piece0 | |
| 15/09/25 20:21:59 INFO storage.BlockManagerInfo: Removed broadcast_8_piece0 on ip-10-69-77-8.ec2.internal:47777 in memory (size: 39.0 KB, free: 1069.9 MB) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerInfo: Removed broadcast_8_piece0 on ip-10-138-58-114.ec2.internal:44188 in memory (size: 39.0 KB, free: 1070.0 MB) | |
| 15/09/25 20:21:59 INFO spark.ContextCleaner: Cleaned broadcast 8 | |
| 15/09/25 20:21:59 INFO storage.BlockManager: Removing broadcast 7 | |
| 15/09/25 20:21:59 INFO storage.BlockManager: Removing block broadcast_7 | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: Block broadcast_7 of size 64936 dropped from memory (free 2221977185) | |
| 15/09/25 20:21:59 INFO storage.BlockManager: Removing block broadcast_7_piece0 | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: Block broadcast_7_piece0 of size 17508 dropped from memory (free 2221994693) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerInfo: Removed broadcast_7_piece0 on ip-10-171-120-202.ec2.internal:50304 in memory (size: 17.1 KB, free: 2.1 GB) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerMaster: Updated info of block broadcast_7_piece0 | |
| 15/09/25 20:21:59 INFO storage.BlockManagerInfo: Removed broadcast_7_piece0 on ip-10-69-77-8.ec2.internal:47777 in memory (size: 17.1 KB, free: 1069.9 MB) | |
| 15/09/25 20:21:59 INFO spark.ContextCleaner: Cleaned broadcast 7 | |
| 15/09/25 20:21:59 INFO storage.BlockManager: Removing broadcast 6 | |
| 15/09/25 20:21:59 INFO storage.BlockManager: Removing block broadcast_6_piece0 | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: Block broadcast_6_piece0 of size 39948 dropped from memory (free 2222034641) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerInfo: Removed broadcast_6_piece0 on ip-10-171-120-202.ec2.internal:50304 in memory (size: 39.0 KB, free: 2.1 GB) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerMaster: Updated info of block broadcast_6_piece0 | |
| 15/09/25 20:21:59 INFO storage.BlockManager: Removing block broadcast_6 | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: Block broadcast_6 of size 432911 dropped from memory (free 2222467552) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerInfo: Removed broadcast_6_piece0 on ip-10-69-77-8.ec2.internal:47777 in memory (size: 39.0 KB, free: 1070.0 MB) | |
| 15/09/25 20:21:59 INFO spark.ContextCleaner: Cleaned broadcast 6 | |
| 15/09/25 20:21:59 INFO storage.BlockManager: Removing broadcast 5 | |
| 15/09/25 20:21:59 INFO storage.BlockManager: Removing block broadcast_5_piece0 | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: Block broadcast_5_piece0 of size 17716 dropped from memory (free 2222485268) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerInfo: Removed broadcast_5_piece0 on ip-10-171-120-202.ec2.internal:50304 in memory (size: 17.3 KB, free: 2.1 GB) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerMaster: Updated info of block broadcast_5_piece0 | |
| 15/09/25 20:21:59 INFO storage.BlockManager: Removing block broadcast_5 | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: Block broadcast_5 of size 64936 dropped from memory (free 2222550204) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerInfo: Removed broadcast_5_piece0 on ip-10-69-77-8.ec2.internal:47777 in memory (size: 17.3 KB, free: 1070.0 MB) | |
| 15/09/25 20:21:59 INFO spark.ContextCleaner: Cleaned broadcast 5 | |
| 15/09/25 20:21:59 INFO storage.BlockManager: Removing broadcast 4 | |
| 15/09/25 20:21:59 INFO storage.BlockManager: Removing block broadcast_4 | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: Block broadcast_4 of size 432911 dropped from memory (free 2222983115) | |
| 15/09/25 20:21:59 INFO storage.BlockManager: Removing block broadcast_4_piece0 | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: Block broadcast_4_piece0 of size 39948 dropped from memory (free 2223023063) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on ip-10-171-120-202.ec2.internal:50304 in memory (size: 39.0 KB, free: 2.1 GB) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerMaster: Updated info of block broadcast_4_piece0 | |
| 15/09/25 20:21:59 INFO storage.BlockManagerInfo: Removed broadcast_4_piece0 on ip-10-69-77-8.ec2.internal:47777 in memory (size: 39.0 KB, free: 1070.0 MB) | |
| 15/09/25 20:21:59 INFO spark.ContextCleaner: Cleaned broadcast 4 | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: ensureFreeSpace(432911) called with curMem=0, maxMem=2223023063 | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: Block broadcast_11 stored as values in memory (estimated size 422.8 KB, free 2.1 GB) | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: ensureFreeSpace(39948) called with curMem=432911, maxMem=2223023063 | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 39.0 KB, free 2.1 GB) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerInfo: Added broadcast_11_piece0 in memory on ip-10-171-120-202.ec2.internal:50304 (size: 39.0 KB, free: 2.1 GB) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerMaster: Updated info of block broadcast_11_piece0 | |
| 15/09/25 20:21:59 INFO spark.SparkContext: Created broadcast 11 from broadcast at TableReader.scala:74 | |
| 15/09/25 20:21:59 INFO mapred.FileInputFormat: Total input paths to process : 10 | |
| 15/09/25 20:21:59 INFO spark.SparkContext: Starting job: runJob at SparkPlan.scala:122 | |
| 15/09/25 20:21:59 INFO scheduler.DAGScheduler: Got job 6 (runJob at SparkPlan.scala:122) with 1 output partitions (allowLocal=false) | |
| 15/09/25 20:21:59 INFO scheduler.DAGScheduler: Final stage: Stage 6(runJob at SparkPlan.scala:122) | |
| 15/09/25 20:21:59 INFO scheduler.DAGScheduler: Parents of final stage: List() | |
| 15/09/25 20:21:59 INFO scheduler.DAGScheduler: Missing parents: List() | |
| 15/09/25 20:21:59 INFO scheduler.DAGScheduler: Submitting Stage 6 (MapPartitionsRDD[29] at map at SparkPlan.scala:97), which has no missing parents | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: ensureFreeSpace(65056) called with curMem=472859, maxMem=2223023063 | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: Block broadcast_12 stored as values in memory (estimated size 63.5 KB, free 2.1 GB) | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: ensureFreeSpace(18068) called with curMem=537915, maxMem=2223023063 | |
| 15/09/25 20:21:59 INFO storage.MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 17.6 KB, free 2.1 GB) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerInfo: Added broadcast_12_piece0 in memory on ip-10-171-120-202.ec2.internal:50304 (size: 17.6 KB, free: 2.1 GB) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerMaster: Updated info of block broadcast_12_piece0 | |
| 15/09/25 20:21:59 INFO spark.SparkContext: Created broadcast 12 from broadcast at DAGScheduler.scala:839 | |
| 15/09/25 20:21:59 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 6 (MapPartitionsRDD[29] at map at SparkPlan.scala:97) | |
| 15/09/25 20:21:59 INFO cluster.YarnScheduler: Adding task set 6.0 with 1 tasks | |
| 15/09/25 20:21:59 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 6.0 (TID 14, ip-10-69-77-8.ec2.internal, NODE_LOCAL, 1354 bytes) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerInfo: Added broadcast_12_piece0 in memory on ip-10-69-77-8.ec2.internal:47777 (size: 17.6 KB, free: 1070.0 MB) | |
| 15/09/25 20:21:59 INFO storage.BlockManagerInfo: Added broadcast_11_piece0 in memory on ip-10-69-77-8.ec2.internal:47777 (size: 39.0 KB, free: 1070.0 MB) | |
| 15/09/25 20:22:05 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 6.0 (TID 14) in 5805 ms on ip-10-69-77-8.ec2.internal (1/1) | |
| 15/09/25 20:22:05 INFO scheduler.DAGScheduler: Stage 6 (runJob at SparkPlan.scala:122) finished in 5.806 s | |
| 15/09/25 20:22:05 INFO cluster.YarnScheduler: Removed TaskSet 6.0, whose tasks have all completed, from pool | |
| 15/09/25 20:22:05 INFO scheduler.DAGScheduler: Job 6 finished: runJob at SparkPlan.scala:122, took 5.817388 s | |
| 15/09/25 20:22:05 INFO spark.SparkContext: Starting job: runJob at SparkPlan.scala:122 | |
| 15/09/25 20:22:05 INFO scheduler.DAGScheduler: Got job 7 (runJob at SparkPlan.scala:122) with 9 output partitions (allowLocal=false) | |
| 15/09/25 20:22:05 INFO scheduler.DAGScheduler: Final stage: Stage 7(runJob at SparkPlan.scala:122) | |
| 15/09/25 20:22:05 INFO scheduler.DAGScheduler: Parents of final stage: List() | |
| 15/09/25 20:22:05 INFO scheduler.DAGScheduler: Missing parents: List() | |
| 15/09/25 20:22:05 INFO scheduler.DAGScheduler: Submitting Stage 7 (MapPartitionsRDD[29] at map at SparkPlan.scala:97), which has no missing parents | |
| 15/09/25 20:22:05 INFO storage.MemoryStore: ensureFreeSpace(65056) called with curMem=555983, maxMem=2223023063 | |
| 15/09/25 20:22:05 INFO storage.MemoryStore: Block broadcast_13 stored as values in memory (estimated size 63.5 KB, free 2.1 GB) | |
| 15/09/25 20:22:05 INFO storage.MemoryStore: ensureFreeSpace(18068) called with curMem=621039, maxMem=2223023063 | |
| 15/09/25 20:22:05 INFO storage.MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 17.6 KB, free 2.1 GB) | |
| 15/09/25 20:22:05 INFO storage.BlockManagerInfo: Added broadcast_13_piece0 in memory on ip-10-171-120-202.ec2.internal:50304 (size: 17.6 KB, free: 2.1 GB) | |
| 15/09/25 20:22:05 INFO storage.BlockManagerMaster: Updated info of block broadcast_13_piece0 | |
| 15/09/25 20:22:05 INFO spark.SparkContext: Created broadcast 13 from broadcast at DAGScheduler.scala:839 | |
| 15/09/25 20:22:05 INFO scheduler.DAGScheduler: Submitting 9 missing tasks from Stage 7 (MapPartitionsRDD[29] at map at SparkPlan.scala:97) | |
| 15/09/25 20:22:05 INFO cluster.YarnScheduler: Adding task set 7.0 with 9 tasks | |
| 15/09/25 20:22:05 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 7.0 (TID 15, ip-10-138-58-114.ec2.internal, NODE_LOCAL, 1354 bytes) | |
| 15/09/25 20:22:05 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 7.0 (TID 16, ip-10-69-77-8.ec2.internal, NODE_LOCAL, 1354 bytes) | |
| 15/09/25 20:22:05 INFO storage.BlockManagerInfo: Added broadcast_13_piece0 in memory on ip-10-69-77-8.ec2.internal:47777 (size: 17.6 KB, free: 1069.9 MB) | |
| 15/09/25 20:22:05 INFO storage.BlockManagerInfo: Added broadcast_13_piece0 in memory on ip-10-138-58-114.ec2.internal:44188 (size: 17.6 KB, free: 1070.0 MB) | |
| 15/09/25 20:22:05 INFO storage.BlockManagerInfo: Added broadcast_11_piece0 in memory on ip-10-138-58-114.ec2.internal:44188 (size: 39.0 KB, free: 1070.0 MB) | |
| 15/09/25 20:22:06 INFO scheduler.TaskSetManager: Starting task 4.0 in stage 7.0 (TID 17, ip-10-138-58-114.ec2.internal, NODE_LOCAL, 1354 bytes) | |
| 15/09/25 20:22:06 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 7.0 (TID 15) in 462 ms on ip-10-138-58-114.ec2.internal (1/9) | |
| 15/09/25 20:22:11 INFO scheduler.TaskSetManager: Starting task 5.0 in stage 7.0 (TID 18, ip-10-69-77-8.ec2.internal, NODE_LOCAL, 1354 bytes) | |
| 15/09/25 20:22:11 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 7.0 (TID 16) in 5706 ms on ip-10-69-77-8.ec2.internal (2/9) | |
| 15/09/25 20:22:11 INFO scheduler.TaskSetManager: Finished task 4.0 in stage 7.0 (TID 17) in 5809 ms on ip-10-138-58-114.ec2.internal (3/9) | |
| 15/09/25 20:22:15 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 7.0 (TID 19, ip-10-138-58-114.ec2.internal, RACK_LOCAL, 1354 bytes) | |
| 15/09/25 20:22:16 INFO metrics.MetricsSaver: Saved 10:30 records to /mnt/var/em/raw/i-661a2299_20150925_SparkSubmit_26404_raw.bin | |
| 15/09/25 20:22:17 INFO scheduler.TaskSetManager: Starting task 7.0 in stage 7.0 (TID 20, ip-10-69-77-8.ec2.internal, NODE_LOCAL, 1354 bytes) | |
| 15/09/25 20:22:17 INFO scheduler.TaskSetManager: Finished task 5.0 in stage 7.0 (TID 18) in 5695 ms on ip-10-69-77-8.ec2.internal (4/9) | |
| 15/09/25 20:22:21 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 7.0 (TID 21, ip-10-138-58-114.ec2.internal, RACK_LOCAL, 1354 bytes) | |
| 15/09/25 20:22:21 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 7.0 (TID 19) in 5784 ms on ip-10-138-58-114.ec2.internal (5/9) | |
| 15/09/25 20:22:23 INFO scheduler.TaskSetManager: Starting task 6.0 in stage 7.0 (TID 22, ip-10-69-77-8.ec2.internal, RACK_LOCAL, 1354 bytes) | |
| 15/09/25 20:22:23 INFO scheduler.TaskSetManager: Finished task 7.0 in stage 7.0 (TID 20) in 6320 ms on ip-10-69-77-8.ec2.internal (6/9) | |
| 15/09/25 20:22:27 INFO scheduler.TaskSetManager: Starting task 8.0 in stage 7.0 (TID 23, ip-10-138-58-114.ec2.internal, RACK_LOCAL, 1354 bytes) | |
| 15/09/25 20:22:27 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 7.0 (TID 21) in 6038 ms on ip-10-138-58-114.ec2.internal (7/9) | |
| 15/09/25 20:22:29 INFO scheduler.TaskSetManager: Finished task 6.0 in stage 7.0 (TID 22) in 5760 ms on ip-10-69-77-8.ec2.internal (8/9) | |
| 15/09/25 20:22:32 INFO scheduler.TaskSetManager: Finished task 8.0 in stage 7.0 (TID 23) in 5853 ms on ip-10-138-58-114.ec2.internal (9/9) | |
| 15/09/25 20:22:32 INFO scheduler.DAGScheduler: Stage 7 (runJob at SparkPlan.scala:122) finished in 27.336 s | |
| 15/09/25 20:22:32 INFO cluster.YarnScheduler: Removed TaskSet 7.0, whose tasks have all completed, from pool | |
| 15/09/25 20:22:32 INFO scheduler.DAGScheduler: Job 7 finished: runJob at SparkPlan.scala:122, took 27.349988 s |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment