Failed to find metadata store when cubing by spark

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Failed to find metadata store when cubing by spark

qiumingming
Hi All:
    I got a error when cubing by spark which make me confused. When I start kylin I got these log:
    2017-06-03 18:09:19,647 INFO  [Thread-8-EventThread] state.ConnectionStateManager:228 : State
    change: CONNECTED
    2017-06-03 18:09:19,680 INFO  [Thread-8] execution.ExecutableManager:53 : Using metadata url:
    kylin_v2_metadata@hbase
   2017-06-03 18:09:19,682 INFO  [Thread-8] dao.ExecutableDao:75 : Using metadata url:
   kylin_v2_metadata@hbase
   2017-06-03 18:09:19,687 INFO  [Thread-8] persistence.ResourceStore:88 : Using metadata url
   kylin_v2_metadata@hbase for resource store
   2017-06-03 18:09:19,708 DEBUG [Thread-8] hbase.HBaseConnection:274 : HTable
   'kylin_v2_metadata' already exists
   2017-06-03 18:09:19,710 DEBUG [Thread-8] metadata.MetadataManager:424 : Reloading SourceTable
   from folder kylin_v2_metadata(key='/table')@kylin_v2_metadata@hbase
   
   It seems the metadata store works well. But when I start cubing by spark , the log file shows these info:
    17/06/03 18:14:16 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, n6-129-021.byted.org): java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_v2_metadata@hbase
        at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99)
        at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:110)
        at org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370)
        at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:298)
        at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109)
        at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
        at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109)
        at org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119)
        at org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467)
        at org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48)
        at org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(AbstractRowKeyEncoder.java:48)
        at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:209)
        at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:197)
        at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1025)
        at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1025)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
        at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
        at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        at org.apache.spark.scheduler.Task.run(Task.scala:89)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:218)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

      I don't know why this happened, I need your help. Thanks.
Reply | Threaded
Open this post in threaded view
|

Re: Failed to find metadata store when cubing by spark

shaofengshi
Spark couldn't connect with the HBase cluster for metadata. Some possible
causes I can think of:
1) the correct hbase-site.xml wasn't in the HADOOP_CONF_DIR
2) some HBase dependent jar wasn't submitted with the application

You may need check more logs, including the Spark executor logs. Start
Spark history sever and then check that job's detail log.

2017-06-03 18:36 GMT+08:00 qiumingming <[hidden email]>:

> Hi All:
>     I got a error when cubing by spark which make me confused. When I start
> kylin I got these log:
>     2017-06-03 18:09:19,647 INFO  [Thread-8-EventThread]
> state.ConnectionStateManager:228 : State
>     change: CONNECTED
>     2017-06-03 18:09:19,680 INFO  [Thread-8] execution.ExecutableManager:53
> : Using metadata url:
>     kylin_v2_metadata@hbase
>    2017-06-03 18:09:19,682 INFO  [Thread-8] dao.ExecutableDao:75 : Using
> metadata url:
>    kylin_v2_metadata@hbase
>    2017-06-03 18:09:19,687 INFO  [Thread-8] persistence.ResourceStore:88 :
> Using metadata url
>    kylin_v2_metadata@hbase for resource store
>    2017-06-03 18:09:19,708 DEBUG [Thread-8] hbase.HBaseConnection:274 :
> HTable
>    'kylin_v2_metadata' already exists
>    2017-06-03 18:09:19,710 DEBUG [Thread-8] metadata.MetadataManager:424 :
> Reloading SourceTable
>    from folder kylin_v2_metadata(key='/table')@kylin_v2_metadata@hbase
>
>    It seems the metadata store works well. But when I start cubing by spark
> , the log file shows these info:
>     17/06/03 18:14:16 WARN scheduler.TaskSetManager: Lost task 0.0 in stage
> 0.0 (TID 0, n6-129-021.byted.org): java.lang.IllegalArgumentException:
> Failed to find metadata store by url: kylin_v2_metadata@hbase
>         at
> org.apache.kylin.common.persistence.ResourceStore.createResourceStore(
> ResourceStore.java:99)
>         at
> org.apache.kylin.common.persistence.ResourceStore.
> getStore(ResourceStore.java:110)
>         at
> org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370)
>         at
> org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(
> CubeDescManager.java:298)
>         at
> org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109)
>         at
> org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
>         at
> org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109)
>         at
> org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119)
>         at
> org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467)
>         at
> org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48)
>         at
> org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(
> AbstractRowKeyEncoder.java:48)
>         at
> org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:209)
>         at
> org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:197)
>         at
> org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(
> JavaPairRDD.scala:1025)
>         at
> org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(
> JavaPairRDD.scala:1025)
>         at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>         at
> org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.
> scala:191)
>         at
> org.apache.spark.shuffle.sort.SortShuffleWriter.write(
> SortShuffleWriter.scala:64)
>         at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
>         at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
>         at org.apache.spark.scheduler.Task.run(Task.scala:89)
>         at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:218)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
>
>       I don't know why this happened, I need your help. Thanks.
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/Failed-to-find-metadata-store-when-cubing-by-spark-tp8167.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>



--
Best regards,

Shaofeng Shi 史少锋
Reply | Threaded
Open this post in threaded view
|

Re: Failed to find metadata store when cubing by spark

qiumingming
(1) the correct hbase-site.xml does in the HADOOP_CONF_DIR.
(2) All dependent jars have been submitted.

Below is the source code of ResourceStore.createResourceStore:
    private static ResourceStore createResourceStore(KylinConfig kylinConfig) {
        List<Throwable> es = new ArrayList<Throwable>();
        logger.info("Using metadata url " + kylinConfig.getMetadataUrl() + " for resource store");
        for (Class<? extends ResourceStore> cls : getKnownImpl(kylinConfig)) {
            try {
                return cls.getConstructor(KylinConfig.class).newInstance(kylinConfig);
            } catch (Throwable e) {
                es.add(e);
            }
        }
        for (Throwable exceptionOrError : es) {
            logger.error("Create new store instance failed ", exceptionOrError);
        }
        throw new IllegalArgumentException("Failed to find metadata store by url: " + kylinConfig.getMetadataUrl());
    }

    This method throw a IllegalArgumentException, and I can see log info like "Failed to find metadata store by url...." in the log file, but I can not search anything which match "Create new store instance failed". This
makes me very confused. If it throw a IllegalArgumentException, why it did not print "Create new store instance failed" in the log file?