Cube failing at building cube stage

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Cube failing at building cube stage

suresh m
Hi All,

Trying to build cube but failing at building cube stage in process. I can
see mapper after more attempts its getting failed.

Sending you the error message, please help me to resolve,

attempt_1493717156723_1006_m_000000_0 100.00 FAILED
hdfs://IALERT2HDP/kylin/kylin_metadata/kylin-ce08d124-99a1-4ee8-aa71-c02c8285dc1e/kylin_intermediate_AL_SFTY_RPT_CUBE_e00df75e_7cdb_4adc_af8d_ecb9db2bc8ab/000000_0:0+214404
> map pm10hdn02.ashokleyland.com:8042 logs
<http://pm10hdn02.ashokleyland.com:8042/node/containerlogs/container_e35_1493717156723_1006_01_000002/hdfs>
Mon
May 15 11:17:53 +0550 2017 Mon May 15 11:21:32 +0550 2017 3mins, 38sec Error:
java.io.IOException: Failed to build cube in mapper 0 at
org.apache.kylin.engine.mr.steps.InMemCuboidMapper.cleanup(InMemCuboidMapper.java:145)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by:
java.util.concurrent.ExecutionException: java.lang.RuntimeException:
java.io.IOException: java.io.IOException: java.lang.RuntimeException:
java.io.IOException: 4 exceptions during in-mem cube build, cause set to
the first, check log for more at
java.util.concurrent.FutureTask.report(FutureTask.java:122) at
java.util.concurrent.FutureTask.get(FutureTask.java:192) at
org.apache.kylin.engine.mr.steps.InMemCuboidMapper.cleanup(InMemCuboidMapper.java:143)
... 8 more Caused by: java.lang.RuntimeException: java.io.IOException:
java.io.IOException: java.lang.RuntimeException: java.io.IOException: 4
exceptions during in-mem cube build, cause set to the first, check log for
more at
org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:82)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException:
java.io.IOException: java.lang.RuntimeException: java.io.IOException: 4
exceptions during in-mem cube build, cause set to the first, check log for
more at
org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:126)
at
org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:73)
at
org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:80)
... 5 more Caused by: java.io.IOException: java.lang.RuntimeException:
java.io.IOException: 4 exceptions during in-mem cube build, cause set to
the first, check log for more at
org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.abort(DoggedCubeBuilder.java:194)
at
org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.checkException(DoggedCubeBuilder.java:167)
at
org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:114)
... 7 more Caused by: java.lang.RuntimeException: java.io.IOException: 4
exceptions during in-mem cube build, cause set to the first, check log for
more at
org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThread.run(DoggedCubeBuilder.java:287)
Caused by: java.io.IOException: 4 exceptions during in-mem cube build,
cause set to the first, check log for more at
org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.throwExceptionIfAny(InMemCubeBuilder.java:228)
at
org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:184)
at
org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.build(InMemCubeBuilder.java:135)
at
org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThread.run(DoggedCubeBuilder.java:282)
Caused by: java.io.IOException: No space left on device at
sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method) at
sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66) at
sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) at
sun.nio.ch.IOUtil.write(IOUtil.java:65) at
sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:745) at
sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:731) at
org.apache.kylin.cube.inmemcubing.ConcurrentDiskStore$Writer$1.write(ConcurrentDiskStore.java:298)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at
java.io.DataOutputStream.flush(DataOutputStream.java:123) at
java.io.FilterOutputStream.close(FilterOutputStream.java:158) at
org.apache.kylin.cube.inmemcubing.ConcurrentDiskStore$Writer.close(ConcurrentDiskStore.java:320)
at org.apache.kylin.gridtable.GTBuilder.close(GTBuilder.java:53) at
org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.scanAndAggregateGridTable(InMemCubeBuilder.java:448)
at
org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.aggregateCuboid(InMemCubeBuilder.java:397)
at
org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.buildCuboid(InMemCubeBuilder.java:389)
at
org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.access$300(InMemCubeBuilder.java:63)
at
org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$CuboidTaskThread.run(InMemCubeBuilder.java:268)
Suppressed: java.io.IOException: No space left on device at
sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method) at
sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66) at
sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) at
sun.nio.ch.IOUtil.write(IOUtil.java:65) at
sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:745) at
sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:731) at
org.apache.kylin.cube.inmemcubing.ConcurrentDiskStore$Writer$1.write(ConcurrentDiskStore.java:298)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at
java.io.FilterOutputStream.close(FilterOutputStream.java:158) at
java.io.FilterOutputStream.close(FilterOutputStream.java:159) ... 7 more
Container killed by the ApplicationMaster. Container killed on request.
Exit code is 143 Container exited with a non-zero exit code 143Regards,
Suresh
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cube failing at building cube stage

shaofengshi
Certain disk is full:

Caused by: java.io.IOException: *No space left on device at*
sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method) at

2017-05-15 15:05 GMT+08:00 suresh m <[hidden email]>:

> Hi All,
>
> Trying to build cube but failing at building cube stage in process. I can
> see mapper after more attempts its getting failed.
>
> Sending you the error message, please help me to resolve,
>
> attempt_1493717156723_1006_m_000000_0 100.00 FAILED
> hdfs://IALERT2HDP/kylin/kylin_metadata/kylin-ce08d124-99a1-
> 4ee8-aa71-c02c8285dc1e/kylin_intermediate_AL_SFTY_RPT_CUBE_
> e00df75e_7cdb_4adc_af8d_ecb9db2bc8ab/000000_0:0+214404
> > map pm10hdn02.ashokleyland.com:8042 logs
> <http://pm10hdn02.ashokleyland.com:8042/node/containerlogs/container_e35_
> 1493717156723_1006_01_000002/hdfs>
> Mon
> May 15 11:17:53 +0550 2017 Mon May 15 11:21:32 +0550 2017 3mins, 38sec
> Error:
> java.io.IOException: Failed to build cube in mapper 0 at
> org.apache.kylin.engine.mr.steps.InMemCuboidMapper.
> cleanup(InMemCuboidMapper.java:145)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1724)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by:
> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> java.io.IOException: java.io.IOException: java.lang.RuntimeException:
> java.io.IOException: 4 exceptions during in-mem cube build, cause set to
> the first, check log for more at
> java.util.concurrent.FutureTask.report(FutureTask.java:122) at
> java.util.concurrent.FutureTask.get(FutureTask.java:192) at
> org.apache.kylin.engine.mr.steps.InMemCuboidMapper.
> cleanup(InMemCuboidMapper.java:143)
> ... 8 more Caused by: java.lang.RuntimeException: java.io.IOException:
> java.io.IOException: java.lang.RuntimeException: java.io.IOException: 4
> exceptions during in-mem cube build, cause set to the first, check log for
> more at
> org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.
> run(AbstractInMemCubeBuilder.java:82)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException:
> java.io.IOException: java.lang.RuntimeException: java.io.IOException: 4
> exceptions during in-mem cube build, cause set to the first, check log for
> more at
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(
> DoggedCubeBuilder.java:126)
> at
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.
> build(DoggedCubeBuilder.java:73)
> at
> org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.
> run(AbstractInMemCubeBuilder.java:80)
> ... 5 more Caused by: java.io.IOException: java.lang.RuntimeException:
> java.io.IOException: 4 exceptions during in-mem cube build, cause set to
> the first, check log for more at
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.abort(
> DoggedCubeBuilder.java:194)
> at
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$
> BuildOnce.checkException(DoggedCubeBuilder.java:167)
> at
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(
> DoggedCubeBuilder.java:114)
> ... 7 more Caused by: java.lang.RuntimeException: java.io.IOException: 4
> exceptions during in-mem cube build, cause set to the first, check log for
> more at
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThread.run(
> DoggedCubeBuilder.java:287)
> Caused by: java.io.IOException: 4 exceptions during in-mem cube build,
> cause set to the first, check log for more at
> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.throwExceptionIfAny(
> InMemCubeBuilder.java:228)
> at
> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.
> build(InMemCubeBuilder.java:184)
> at
> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.
> build(InMemCubeBuilder.java:135)
> at
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$SplitThread.run(
> DoggedCubeBuilder.java:282)
> Caused by: java.io.IOException: No space left on device at
> sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method) at
> sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66) at
> sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) at
> sun.nio.ch.IOUtil.write(IOUtil.java:65) at
> sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:745) at
> sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:731) at
> org.apache.kylin.cube.inmemcubing.ConcurrentDiskStore$Writer$1.
> write(ConcurrentDiskStore.java:298)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at
> java.io.DataOutputStream.flush(DataOutputStream.java:123) at
> java.io.FilterOutputStream.close(FilterOutputStream.java:158) at
> org.apache.kylin.cube.inmemcubing.ConcurrentDiskStore$Writer.
> close(ConcurrentDiskStore.java:320)
> at org.apache.kylin.gridtable.GTBuilder.close(GTBuilder.java:53) at
> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.
> scanAndAggregateGridTable(InMemCubeBuilder.java:448)
> at
> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.aggregateCuboid(
> InMemCubeBuilder.java:397)
> at
> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.
> buildCuboid(InMemCubeBuilder.java:389)
> at
> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder.
> access$300(InMemCubeBuilder.java:63)
> at
> org.apache.kylin.cube.inmemcubing.InMemCubeBuilder$CuboidTaskThread.run(
> InMemCubeBuilder.java:268)
> Suppressed: java.io.IOException: No space left on device at
> sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method) at
> sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66) at
> sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) at
> sun.nio.ch.IOUtil.write(IOUtil.java:65) at
> sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:745) at
> sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:731) at
> org.apache.kylin.cube.inmemcubing.ConcurrentDiskStore$Writer$1.
> write(ConcurrentDiskStore.java:298)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at
> java.io.FilterOutputStream.close(FilterOutputStream.java:158) at
> java.io.FilterOutputStream.close(FilterOutputStream.java:159) ... 7 more
> Container killed by the ApplicationMaster. Container killed on request.
> Exit code is 143 Container exited with a non-zero exit code 143Regards,
> Suresh
>



--
Best regards,

Shaofeng Shi 史少锋
Loading...