It does not work use Kylin-2.0-spark-1.5.2-hadoop-2.7

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

It does not work use Kylin-2.0-spark-1.5.2-hadoop-2.7

hger
System: Kylin-2.0 spark-cluster-1.5.2 HDP2.3:hadoop-2.7.1.2 lib spark-assembly-1.5.2.2.3.4.7.4-hadoop2.7.1.2.3.4.7.jar

Error occured when I build the cube with kylin-engine while the job has added to the kylin-history webUI

17/05/23 17:26:04 INFO CubeManager: Reloaded cube DFAD being CUBE[name=DFAD] having 0 segments
17/05/23 17:26:04 INFO CubeManager: Reloaded cube kylin_sales_cube being CUBE[name=kylin_sales_cube] having 1 segments
17/05/23 17:26:04 INFO CubeManager: Loaded 2 cubes, fail on 0 cubes
17/05/23 17:26:04 INFO MemoryStore: ensureFreeSpace(99864) called with curMem=0, maxMem=556038881
17/05/23 17:26:04 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 97.5 KB, free 530.2 MB)
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/execution/columnar/CachedBatch
        at org.apache.kylin.engine.spark.KylinKryoRegistrator.registerClasses(KylinKryoRegistrator.java:70)
        at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$4.apply(KryoSerializer.scala:120)
        at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$4.apply(KryoSerializer.scala:120)
        at scala.Option.foreach(Option.scala:236)
        at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:120)
        at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:237)
        at org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:222)
        at org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:138)
        at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:201)
        at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:102)
        at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:85)
        at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
        at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
        at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1327)
        at org.apache.spark.api.java.JavaSparkContext.broadcast(JavaSparkContext.scala:648)
        at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:166)
        at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
        at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:685)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.execution.columnar.CachedBatch
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 27 more
17/05/23 17:26:04 INFO ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
17/05/23 17:26:04 INFO SparkContext: Invoking stop() from shutdown hook
17/05/23 17:26:04 INFO ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x15c204d83a0000b

Could you help me? Thand you!
Reply | Threaded
Open this post in threaded view
|

Re: It does not work use Kylin-2.0-spark-1.5.2-hadoop-2.7

shaofengshi
Yes Kylin 2.0 is compiled with Spark 1.6.3, so please use the embeded Spark
in Kylin binary. Using other versions may got such mis-matched errors.

2017-05-23 18:15 GMT+08:00 hger <[hidden email]>:

> System: Kylin-2.0 spark-cluster-1.5.2 HDP2.3:hadoop-2.7.1.2 lib
> spark-assembly-1.5.2.2.3.4.7.4-hadoop2.7.1.2.3.4.7.jar
>
> Error occured when I build the cube with kylin-engine while the job has
> added to the kylin-history webUI
>
> 17/05/23 17:26:04 INFO CubeManager: Reloaded cube DFAD being
> CUBE[name=DFAD]
> having 0 segments
> 17/05/23 17:26:04 INFO CubeManager: Reloaded cube kylin_sales_cube being
> CUBE[name=kylin_sales_cube] having 1 segments
> 17/05/23 17:26:04 INFO CubeManager: Loaded 2 cubes, fail on 0 cubes
> 17/05/23 17:26:04 INFO MemoryStore: ensureFreeSpace(99864) called with
> curMem=0, maxMem=556038881
> 17/05/23 17:26:04 INFO MemoryStore: Block broadcast_0 stored as values in
> memory (estimated size 97.5 KB, free 530.2 MB)
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/spark/sql/execution/columnar/CachedBatch
>         at
> org.apache.kylin.engine.spark.KylinKryoRegistrator.registerClasses(
> KylinKryoRegistrator.java:70)
>         at
> org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$4.apply(
> KryoSerializer.scala:120)
>         at
> org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$4.apply(
> KryoSerializer.scala:120)
>         at scala.Option.foreach(Option.scala:236)
>         at
> org.apache.spark.serializer.KryoSerializer.newKryo(
> KryoSerializer.scala:120)
>         at
> org.apache.spark.serializer.KryoSerializerInstance.
> borrowKryo(KryoSerializer.scala:237)
>         at
> org.apache.spark.serializer.KryoSerializerInstance.<init>(
> KryoSerializer.scala:222)
>         at
> org.apache.spark.serializer.KryoSerializer.newInstance(
> KryoSerializer.scala:138)
>         at
> org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(
> TorrentBroadcast.scala:201)
>         at
> org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(
> TorrentBroadcast.scala:102)
>         at
> org.apache.spark.broadcast.TorrentBroadcast.<init>(
> TorrentBroadcast.scala:85)
>         at
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(
> TorrentBroadcastFactory.scala:34)
>         at
> org.apache.spark.broadcast.BroadcastManager.newBroadcast(
> BroadcastManager.scala:63)
>         at org.apache.spark.SparkContext.broadcast(SparkContext.scala:
> 1327)
>         at
> org.apache.spark.api.java.JavaSparkContext.broadcast(
> JavaSparkContext.scala:648)
>         at
> org.apache.kylin.engine.spark.SparkCubingByLayer.execute(
> SparkCubingByLayer.java:166)
>         at
> org.apache.kylin.common.util.AbstractApplication.execute(
> AbstractApplication.java:37)
>         at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.
> java:44)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$
> deploy$SparkSubmit$$runMain(SparkSubmit.scala:685)
>         at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(
> SparkSubmit.scala:180)
>         at org.apache.spark.deploy.SparkSubmit$.submit(
> SparkSubmit.scala:205)
>         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.
> scala:120)
>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.spark.sql.execution.columnar.CachedBatch
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>         ... 27 more
> 17/05/23 17:26:04 INFO ConnectionManager$HConnectionImplementation:
> Closing
> master protocol: MasterService
> 17/05/23 17:26:04 INFO SparkContext: Invoking stop() from shutdown hook
> 17/05/23 17:26:04 INFO ConnectionManager$HConnectionImplementation:
> Closing
> zookeeper sessionid=0x15c204d83a0000b
>
> Could you help me? Thand you!
>
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/It-does-not-work-use-Kylin-2-0-spark-1-5-2-
> hadoop-2-7-tp8073.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>



--
Best regards,

Shaofeng Shi 史少锋