Skip to content

Instantly share code, notes, and snippets.

@GUI

GUI/kylin_null_sample_build.log

Created March 23, 2016 04:35
Download ZIP

Revisions

  1. GUI created this gist Mar 23, 2016.
    139 changes: 139 additions & 0 deletions kylin_null_sample_build.log
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,139 @@
    2016-03-22 19:53:07.314 - State of Hadoop job: job_1458515807807_0318:ACCEPTED - UNDEFINED
    2016-03-22 19:53:17.320 - State of Hadoop job: job_1458515807807_0318:RUNNING - UNDEFINED
    2016-03-22 19:53:27.328 - State of Hadoop job: job_1458515807807_0318:RUNNING - UNDEFINED
    2016-03-22 19:53:37.334 - State of Hadoop job: job_1458515807807_0318:RUNNING - UNDEFINED
    2016-03-22 19:53:47.339 - State of Hadoop job: job_1458515807807_0318:RUNNING - UNDEFINED
    2016-03-22 19:53:57.344 - State of Hadoop job: job_1458515807807_0318:RUNNING - UNDEFINED
    2016-03-22 19:54:07.351 - State of Hadoop job: job_1458515807807_0318:RUNNING - UNDEFINED
    2016-03-22 19:54:17.356 - State of Hadoop job: job_1458515807807_0318:FINISHED - FAILED
    no counters for job job_1458515807807_0318

    2016-03-23 02:53:15,103 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties
    2016-03-23 02:53:15,194 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
    2016-03-23 02:53:15,194 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
    2016-03-23 02:53:15,206 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
    2016-03-23 02:53:15,266 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1458515807807_0318, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@207fd4ec)
    2016-03-23 02:53:15,299 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
    2016-03-23 02:53:15,544 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /hadoop/yarn/local/usercache/hdfs/appcache/application_1458515807807_0318
    2016-03-23 02:53:15,987 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
    2016-03-23 02:53:16,391 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
    2016-03-23 02:53:16,412 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ]
    2016-03-23 02:53:16,671 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: org.apache.hive.hcatalog.mapreduce.HCatSplit@2d7d97e5
    2016-03-23 02:53:16,760 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 52428796(209715184)
    2016-03-23 02:53:16,760 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 200
    2016-03-23 02:53:16,760 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at 146800640
    2016-03-23 02:53:16,760 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 209715200
    2016-03-23 02:53:16,760 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 52428796; length = 13107200
    2016-03-23 02:53:16,767 INFO [main] org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
    2016-03-23 02:53:16,792 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.snappy]
    2016-03-23 02:53:16,796 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.snappy]
    2016-03-23 02:53:16,797 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.snappy]
    2016-03-23 02:53:16,797 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor [.snappy]
    2016-03-23 02:53:16,808 INFO [main] org.apache.hive.hcatalog.mapreduce.InternalUtil: Initializing org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe with properties {name=default.kylin_intermediate_logs_test_cube_20160110000000_20160316000000, numFiles=1, field.delim=, columns.types=date,int,string, serialization.format=, columns=default_logs_test_cal_date,default_logs_test_response_time,default_logs_test_denied_reason, rawDataSize=170, numRows=10, EXTERNAL=TRUE, serialization.lib=org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, COLUMN_STATS_ACCURATE=true, totalSize=291, serialization.null.format=\N, transient_lastDdlTime=1458701552}
    2016-03-23 02:53:16,902 INFO [main] org.apache.kylin.engine.mr.common.AbstractHadoopJob: The absolute path for meta dir is /mnt/hadoop/yarn/local/usercache/hdfs/appcache/application_1458515807807_0318/container_e11_1458515807807_0318_01_000002/meta
    2016-03-23 02:53:16,905 INFO [main] org.apache.kylin.common.KylinConfig: Use KYLIN_CONF=/mnt/hadoop/yarn/local/usercache/hdfs/appcache/application_1458515807807_0318/container_e11_1458515807807_0318_01_000002/meta
    2016-03-23 02:53:16,908 INFO [main] org.apache.kylin.common.KylinConfigBase: Kylin Config was updated with kylin.metadata.url : /mnt/hadoop/yarn/local/usercache/hdfs/appcache/application_1458515807807_0318/container_e11_1458515807807_0318_01_000002/meta
    2016-03-23 02:53:16,921 INFO [main] org.apache.kylin.cube.CubeManager: Initializing CubeManager with config /mnt/hadoop/yarn/local/usercache/hdfs/appcache/application_1458515807807_0318/container_e11_1458515807807_0318_01_000002/meta
    2016-03-23 02:53:16,924 INFO [main] org.apache.kylin.common.persistence.ResourceStore: Using metadata url /mnt/hadoop/yarn/local/usercache/hdfs/appcache/application_1458515807807_0318/container_e11_1458515807807_0318_01_000002/meta for resource store
    2016-03-23 02:53:17,045 WARN [main] org.apache.kylin.common.persistence.ResourceStore: Failed to load ResourceStore impl class: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/filter/Filter
    2016-03-23 02:53:17,269 INFO [main] org.apache.kylin.cube.CubeDescManager: Initializing CubeDescManager with config /mnt/hadoop/yarn/local/usercache/hdfs/appcache/application_1458515807807_0318/container_e11_1458515807807_0318_01_000002/meta
    2016-03-23 02:53:17,269 INFO [main] org.apache.kylin.cube.CubeDescManager: Reloading Cube Metadata from folder /mnt/hadoop/yarn/local/usercache/hdfs/appcache/application_1458515807807_0318/container_e11_1458515807807_0318_01_000002/meta/cube_desc
    2016-03-23 02:53:17,394 INFO [main] org.apache.kylin.dict.DictionaryManager: DictionaryManager(1310836610) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.LOGS_TEST/CAL_DATE/48ba5e36-50b4-4d54-a6a3-c2d182eef786.dict
    2016-03-23 02:53:17,401 INFO [main] org.apache.kylin.dict.DictionaryManager: DictionaryManager(1310836610) loading DictionaryInfo(loadDictObj:true) at /dict/DEFAULT.LOGS_TEST/RESPONSE_TIME/0c4c3f2a-a1e8-4c60-8ef4-2481185ecfbd.dict
    2016-03-23 02:53:17,405 WARN [main] org.apache.kylin.engine.mr.steps.InMemCuboidMapper: Dictionary for DEFAULT.LOGS_TEST.DENIED_REASON was not found.
    2016-03-23 02:53:17,407 INFO [main] org.apache.kylin.engine.mr.steps.InMemCuboidMapper: Reserve 432 MB = 200 (MR reserve) + 232 (SYS reserve)
    2016-03-23 02:53:17,412 INFO [pool-5-thread-1] org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Dogged Cube Build start
    2016-03-23 02:53:17,416 INFO [pool-5-thread-1] org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Split #1 kickoff
    2016-03-23 02:53:17,417 INFO [Thread-11] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: In Mem Cube Build start, logs_test_cube
    2016-03-23 02:53:17,417 INFO [main] org.apache.kylin.engine.mr.steps.InMemCuboidMapper: Totally handled 10 records!
    2016-03-23 02:53:19,589 INFO [Thread-11] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: Calculating base cuboid 7
    2016-03-23 02:53:19,615 INFO [Thread-11] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: Cuboid 7 has 10 rows, build takes 26ms
    2016-03-23 02:53:19,615 INFO [Thread-11] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: Wild estimate of base aggr cache is 0 MB
    2016-03-23 02:53:19,616 INFO [Thread-11] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: collecting CuboidResult cuboid id:7
    2016-03-23 02:53:27,006 INFO [CuboidTask-2] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: Calculating cuboid 6
    2016-03-23 02:53:27,006 INFO [CuboidTask-1] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: Calculating cuboid 5
    2016-03-23 02:53:27,006 INFO [CuboidTask-0] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: Calculating cuboid 3
    2016-03-23 02:53:27,011 INFO [CuboidTask-0] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: Cuboid 3 has 2 rows, build takes 5ms
    2016-03-23 02:53:27,011 INFO [CuboidTask-0] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: collecting CuboidResult cuboid id:3
    2016-03-23 02:53:27,011 INFO [CuboidTask-1] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: Cuboid 5 has 10 rows, build takes 5ms
    2016-03-23 02:53:27,011 INFO [CuboidTask-2] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: Cuboid 6 has 10 rows, build takes 5ms
    2016-03-23 02:53:27,011 INFO [CuboidTask-0] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: Calculating cuboid 1
    2016-03-23 02:53:27,011 INFO [CuboidTask-2] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: collecting CuboidResult cuboid id:6
    2016-03-23 02:53:27,011 INFO [CuboidTask-1] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: collecting CuboidResult cuboid id:5
    2016-03-23 02:53:27,012 INFO [CuboidTask-1] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: Calculating cuboid 4
    2016-03-23 02:53:27,012 INFO [CuboidTask-0] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: Cuboid 1 has 1 rows, build takes 1ms
    2016-03-23 02:53:27,012 INFO [CuboidTask-0] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: collecting CuboidResult cuboid id:1
    2016-03-23 02:53:27,011 INFO [CuboidTask-3] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: Calculating cuboid 2
    2016-03-23 02:53:27,013 INFO [CuboidTask-1] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: Cuboid 4 has 10 rows, build takes 1ms
    2016-03-23 02:53:27,013 INFO [CuboidTask-1] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: collecting CuboidResult cuboid id:4
    2016-03-23 02:53:27,014 INFO [CuboidTask-3] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: Cuboid 2 has 2 rows, build takes 3ms
    2016-03-23 02:53:27,014 INFO [CuboidTask-3] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: collecting CuboidResult cuboid id:2
    2016-03-23 02:53:27,015 INFO [Thread-11] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: In Mem Cube Build end, logs_test_cube, takes 9598 ms
    2016-03-23 02:53:27,015 INFO [Thread-11] org.apache.kylin.cube.inmemcubing.InMemCubeBuilder: total CuboidResult count:7
    2016-03-23 02:53:27,015 INFO [pool-5-thread-1] org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Dogged Cube Build splits complete, took 9603 ms
    2016-03-23 02:53:27,018 ERROR [pool-5-thread-1] org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Dogged Cube Build error
    java.lang.ArrayIndexOutOfBoundsException
    at java.lang.System.arraycopy(Native Method)
    at org.apache.kylin.gridtable.GTRecord.exportColumns(GTRecord.java:253)
    at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:84)
    at org.apache.kylin.engine.mr.steps.KVGTRecordWriter.write(KVGTRecordWriter.java:76)
    at org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder.outputCuboid(AbstractInMemCubeBuilder.java:89)
    at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$Merger.mergeAndOutput(DoggedCubeBuilder.java:308)
    at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:116)
    at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:72)
    at org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:74)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
    2016-03-23 02:53:27,019 ERROR [pool-5-thread-1] org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Error closing grid table org.apache.kylin.gridtable.GridTable@6da87752
    java.lang.IllegalStateException
    at org.apache.kylin.cube.inmemcubing.ConcurrentDiskStore.close(ConcurrentDiskStore.java:357)
    at org.apache.kylin.gridtable.GridTable.close(GridTable.java:66)
    at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.closeGirdTables(DoggedCubeBuilder.java:140)
    at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:128)
    at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:72)
    at org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:74)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
    2016-03-23 02:53:27,020 INFO [pool-5-thread-1] org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Dogged Cube Build end, totally took 9608 ms
    2016-03-23 02:53:27,020 INFO [pool-5-thread-1] org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder: Dogged Cube Build return
    2016-03-23 02:53:27,020 INFO [main] org.apache.hadoop.mapred.MapTask: Starting flush of map output
    2016-03-23 02:53:27,033 INFO [main] org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor [.snappy]
    2016-03-23 02:53:27,036 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Failed to build cube in mapper 0
    at org.apache.kylin.engine.mr.steps.InMemCuboidMapper.cleanup(InMemCuboidMapper.java:142)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:148)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
    Caused by: java.util.concurrent.ExecutionException: java.lang.ArrayIndexOutOfBoundsException
    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.util.concurrent.FutureTask.get(FutureTask.java:188)
    at org.apache.kylin.engine.mr.steps.InMemCuboidMapper.cleanup(InMemCuboidMapper.java:140)
    ... 8 more
    Caused by: java.lang.ArrayIndexOutOfBoundsException
    at java.lang.System.arraycopy(Native Method)
    at org.apache.kylin.gridtable.GTRecord.exportColumns(GTRecord.java:253)
    at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:84)
    at org.apache.kylin.engine.mr.steps.KVGTRecordWriter.write(KVGTRecordWriter.java:76)
    at org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder.outputCuboid(AbstractInMemCubeBuilder.java:89)
    at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$Merger.mergeAndOutput(DoggedCubeBuilder.java:308)
    at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:116)
    at org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:72)
    at org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:74)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

    2016-03-23 02:53:27,039 INFO [main] org.apache.hadoop.mapred.Task: Runnning cleanup for the task
    2016-03-23 02:53:27,043 WARN [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://ip-10-0-20-142.ec2.internal:8020/kylin/kylin_metadata/kylin-2d8ad2a3-c7f1-4436-84ed-4473892c9a95/logs_test_cube/cuboid/_temporary/1/_temporary/attempt_1458515807807_0318_m_000000_0