Re: File does not exist: hdfs:/kylin/spark-history

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: File does not exist: hdfs:/kylin/spark-history

shaofengshi
It was a configuration in kylin.properties:

kylin.engine.spark-conf.spark.eventLog.dir=hdfs\:///kylin/spark-history
kylin.engine.spark-conf.spark.history.fs.logDirectory=hdfs\:///kylin/spark-history


2017-06-02 16:47 GMT+08:00 qiumingming <[hidden email]>:

> Hi All, I got a error at Build Cube with Spark step, below is my error
> info.
> I can not search anything which match "hdfs:/kylin/spark-history" in the
> project source code:
>         ERROR spark.SparkContext: Error initializing SparkContext.
>         java.io.FileNotFoundException: File does not exist:
> hdfs:/kylin/spark-history
>         at
> org.apache.hadoop.hdfs.DistributedFileSystem$18.
> doCall(DistributedFileSystem.java:1132)
>         at
> org.apache.hadoop.hdfs.DistributedFileSystem$18.
> doCall(DistributedFileSystem.java:1124)
>         at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(
> FileSystemLinkResolver.java:81)
>         at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(
> DistributedFileSystem.java:1124)
>         at
> org.apache.spark.scheduler.EventLoggingListener.start(
> EventLoggingListener.scala:100)
>         at org.apache.spark.SparkContext.<init>(SparkContext.scala:583)
>         at
> org.apache.spark.api.java.JavaSparkContext.<init>(
> JavaSparkContext.scala:59)
>         at
> org.apache.kylin.engine.spark.SparkCubingByLayer.execute(
> SparkCubingByLayer.java:154)
>         at
> org.apache.kylin.common.util.AbstractApplication.execute(
> AbstractApplication.java:37)
>         at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.
> java:44)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 62)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$
> deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
>         at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(
> SparkSubmit.scala:181)
>         at org.apache.spark.deploy.SparkSubmit$.submit(
> SparkSubmit.scala:206)
>         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.
> scala:121)
>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/File-does-not-exist-hdfs-kylin-spark-history-tp8154.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>



--
Best regards,

Shaofeng Shi 史少锋
Reply | Threaded
Open this post in threaded view
|

Re: File does not exist: hdfs:/kylin/spark-history

qiumingming
I have solved this problem yesterday. Thank you all the same.
Reply | Threaded
Open this post in threaded view
|

Re: File does not exist: hdfs:/kylin/spark-history

shaofengshi
Cool, how about the Spark engine's performance in your scenario?

2017-06-03 17:20 GMT+08:00 qiumingming <[hidden email]>:

> I have solved this problem yesterday. Thank you all the same.
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/Re-File-does-not-exist-hdfs-kylin-spark-
> history-tp8159p8164.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>



--
Best regards,

Shaofeng Shi 史少锋
Reply | Threaded
Open this post in threaded view
|

Re: File does not exist: hdfs:/kylin/spark-history

qiumingming
I steel have error when cubing with spark. I am working on it, hope it can be solved today.