Skip to content

Instantly share code, notes, and snippets.

@vmakhaev

vmakhaev/Spark cached query logs

Created September 25, 2015 21:35
Show Gist options
  • Save vmakhaev/38a1da9a6af8c92f1577 to your computer and use it in GitHub Desktop.
Save vmakhaev/38a1da9a6af8c92f1577 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Spark cached query logs
scala> hc.sql("select * from price_equities_technicals where marketdate = 1443052800000 and sid = 21438 limit 1").show()
15/09/25 21:35:09 INFO parse.ParseDriver: Parsing command: select * from price_equities_technicals where marketdate = 1443052800000 and sid = 21438 limit 1
15/09/25 21:35:09 INFO parse.ParseDriver: Parse Completed
15/09/25 21:35:09 INFO columnar.InMemoryColumnarTableScan: Predicate (marketdate#0L = 1443052800000) generates partition filter: ((marketdate.lowerBound#3073L <= 1443052800000) && (1443052800000 <= marketdate.upperBound#3072L))
15/09/25 21:35:09 INFO columnar.InMemoryColumnarTableScan: Predicate (sid#1 = 21438) generates partition filter: ((sid.lowerBound#3078 <= 21438) && (21438 <= sid.upperBound#3077))
15/09/25 21:35:09 INFO spark.SparkContext: Starting job: runJob at SparkPlan.scala:122
15/09/25 21:35:09 INFO scheduler.DAGScheduler: Got job 10 (runJob at SparkPlan.scala:122) with 1 output partitions (allowLocal=false)
15/09/25 21:35:09 INFO scheduler.DAGScheduler: Final stage: Stage 10(runJob at SparkPlan.scala:122)
15/09/25 21:35:09 INFO scheduler.DAGScheduler: Parents of final stage: List()
15/09/25 21:35:09 INFO scheduler.DAGScheduler: Missing parents: List()
15/09/25 21:35:09 INFO scheduler.DAGScheduler: Submitting Stage 10 (MapPartitionsRDD[27] at map at SparkPlan.scala:97), which has no missing parents
15/09/25 21:35:09 INFO storage.MemoryStore: ensureFreeSpace(262600) called with curMem=1140274, maxMem=2223023063
15/09/25 21:35:09 INFO storage.MemoryStore: Block broadcast_14 stored as values in memory (estimated size 256.4 KB, free 2.1 GB)
15/09/25 21:35:09 INFO storage.MemoryStore: ensureFreeSpace(71039) called with curMem=1402874, maxMem=2223023063
15/09/25 21:35:09 INFO storage.MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 69.4 KB, free 2.1 GB)
15/09/25 21:35:09 INFO storage.BlockManagerInfo: Added broadcast_14_piece0 in memory on ip-10-171-120-202.ec2.internal:52391 (size: 69.4 KB, free: 2.1 GB)
15/09/25 21:35:09 INFO storage.BlockManagerMaster: Updated info of block broadcast_14_piece0
15/09/25 21:35:09 INFO spark.SparkContext: Created broadcast 14 from broadcast at DAGScheduler.scala:839
15/09/25 21:35:09 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 10 (MapPartitionsRDD[27] at map at SparkPlan.scala:97)
15/09/25 21:35:09 INFO cluster.YarnScheduler: Adding task set 10.0 with 1 tasks
15/09/25 21:35:09 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 10.0 (TID 50, ip-10-218-180-157.ec2.internal, PROCESS_LOCAL, 1354 bytes)
15/09/25 21:35:09 INFO storage.BlockManagerInfo: Added broadcast_14_piece0 in memory on ip-10-218-180-157.ec2.internal:38115 (size: 69.4 KB, free: 765.9 MB)
15/09/25 21:35:10 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 10.0 (TID 50) in 230 ms on ip-10-218-180-157.ec2.internal (1/1)
15/09/25 21:35:10 INFO cluster.YarnScheduler: Removed TaskSet 10.0, whose tasks have all completed, from pool
15/09/25 21:35:10 INFO scheduler.DAGScheduler: Stage 10 (runJob at SparkPlan.scala:122) finished in 0.230 s
15/09/25 21:35:10 INFO scheduler.DAGScheduler: Job 10 finished: runJob at SparkPlan.scala:122, took 0.250311 s
15/09/25 21:35:10 INFO spark.SparkContext: Starting job: runJob at SparkPlan.scala:122
15/09/25 21:35:10 INFO scheduler.DAGScheduler: Got job 11 (runJob at SparkPlan.scala:122) with 9 output partitions (allowLocal=false)
15/09/25 21:35:10 INFO scheduler.DAGScheduler: Final stage: Stage 11(runJob at SparkPlan.scala:122)
15/09/25 21:35:10 INFO scheduler.DAGScheduler: Parents of final stage: List()
15/09/25 21:35:10 INFO scheduler.DAGScheduler: Missing parents: List()
15/09/25 21:35:10 INFO scheduler.DAGScheduler: Submitting Stage 11 (MapPartitionsRDD[27] at map at SparkPlan.scala:97), which has no missing parents
15/09/25 21:35:10 INFO storage.MemoryStore: ensureFreeSpace(262600) called with curMem=1473913, maxMem=2223023063
15/09/25 21:35:10 INFO storage.MemoryStore: Block broadcast_15 stored as values in memory (estimated size 256.4 KB, free 2.1 GB)
15/09/25 21:35:10 INFO storage.MemoryStore: ensureFreeSpace(71039) called with curMem=1736513, maxMem=2223023063
15/09/25 21:35:10 INFO storage.MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 69.4 KB, free 2.1 GB)
15/09/25 21:35:10 INFO storage.BlockManagerInfo: Added broadcast_15_piece0 in memory on ip-10-171-120-202.ec2.internal:52391 (size: 69.4 KB, free: 2.1 GB)
15/09/25 21:35:10 INFO storage.BlockManagerMaster: Updated info of block broadcast_15_piece0
15/09/25 21:35:10 INFO spark.SparkContext: Created broadcast 15 from broadcast at DAGScheduler.scala:839
15/09/25 21:35:10 INFO scheduler.DAGScheduler: Submitting 9 missing tasks from Stage 11 (MapPartitionsRDD[27] at map at SparkPlan.scala:97)
15/09/25 21:35:10 INFO cluster.YarnScheduler: Adding task set 11.0 with 9 tasks
15/09/25 21:35:10 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 11.0 (TID 51, ip-10-155-165-230.ec2.internal, PROCESS_LOCAL, 1354 bytes)
15/09/25 21:35:10 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 11.0 (TID 52, ip-10-218-180-157.ec2.internal, PROCESS_LOCAL, 1354 bytes)
15/09/25 21:35:10 INFO storage.BlockManagerInfo: Added broadcast_15_piece0 in memory on ip-10-155-165-230.ec2.internal:53879 (size: 69.4 KB, free: 867.2 MB)
15/09/25 21:35:10 INFO storage.BlockManagerInfo: Added broadcast_15_piece0 in memory on ip-10-218-180-157.ec2.internal:38115 (size: 69.4 KB, free: 765.8 MB)
15/09/25 21:35:10 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 11.0 (TID 53, ip-10-218-180-157.ec2.internal, PROCESS_LOCAL, 1354 bytes)
15/09/25 21:35:10 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 11.0 (TID 52) in 36 ms on ip-10-218-180-157.ec2.internal (1/9)
15/09/25 21:35:10 INFO scheduler.TaskSetManager: Starting task 5.0 in stage 11.0 (TID 54, ip-10-155-165-230.ec2.internal, PROCESS_LOCAL, 1354 bytes)
15/09/25 21:35:10 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 11.0 (TID 51) in 237 ms on ip-10-155-165-230.ec2.internal (2/9)
15/09/25 21:35:10 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 11.0 (TID 55, ip-10-218-180-157.ec2.internal, PROCESS_LOCAL, 1354 bytes)
15/09/25 21:35:10 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 11.0 (TID 53) in 231 ms on ip-10-218-180-157.ec2.internal (3/9)
15/09/25 21:35:10 INFO scheduler.TaskSetManager: Starting task 6.0 in stage 11.0 (TID 56, ip-10-155-165-230.ec2.internal, PROCESS_LOCAL, 1354 bytes)
15/09/25 21:35:10 INFO scheduler.TaskSetManager: Finished task 5.0 in stage 11.0 (TID 54) in 223 ms on ip-10-155-165-230.ec2.internal (4/9)
15/09/25 21:35:10 INFO scheduler.TaskSetManager: Starting task 4.0 in stage 11.0 (TID 57, ip-10-218-180-157.ec2.internal, PROCESS_LOCAL, 1354 bytes)
15/09/25 21:35:10 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 11.0 (TID 55) in 227 ms on ip-10-218-180-157.ec2.internal (5/9)
15/09/25 21:35:10 INFO scheduler.TaskSetManager: Starting task 7.0 in stage 11.0 (TID 58, ip-10-155-165-230.ec2.internal, PROCESS_LOCAL, 1354 bytes)
15/09/25 21:35:10 INFO scheduler.TaskSetManager: Finished task 6.0 in stage 11.0 (TID 56) in 218 ms on ip-10-155-165-230.ec2.internal (6/9)
15/09/25 21:35:10 INFO scheduler.TaskSetManager: Starting task 8.0 in stage 11.0 (TID 59, ip-10-218-180-157.ec2.internal, PROCESS_LOCAL, 1354 bytes)
15/09/25 21:35:10 INFO scheduler.TaskSetManager: Finished task 4.0 in stage 11.0 (TID 57) in 216 ms on ip-10-218-180-157.ec2.internal (7/9)
15/09/25 21:35:11 INFO scheduler.TaskSetManager: Finished task 7.0 in stage 11.0 (TID 58) in 216 ms on ip-10-155-165-230.ec2.internal (8/9)
15/09/25 21:35:11 INFO scheduler.TaskSetManager: Finished task 8.0 in stage 11.0 (TID 59) in 214 ms on ip-10-218-180-157.ec2.internal (9/9)
15/09/25 21:35:11 INFO cluster.YarnScheduler: Removed TaskSet 11.0, whose tasks have all completed, from pool
15/09/25 21:35:11 INFO scheduler.DAGScheduler: Stage 11 (runJob at SparkPlan.scala:122) finished in 0.921 s
15/09/25 21:35:11 INFO scheduler.DAGScheduler: Job 11 finished: runJob at SparkPlan.scala:122, took 0.950625 s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment