[jira] [Created] (KYLIN-2651) Couldn't find global dict index file

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Created] (KYLIN-2651) Couldn't find global dict index file

JIRA jira@apache.org
Shaofeng SHI created KYLIN-2651:
-----------------------------------

             Summary: Couldn't find global dict index file
                 Key: KYLIN-2651
                 URL: https://issues.apache.org/jira/browse/KYLIN-2651
             Project: Kylin
          Issue Type: Bug
            Reporter: Shaofeng SHI


Today I got an error when run a tes cube with Spark engine:

{code}

17/05/31 08:50:28 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, sandbox): java.lang.RuntimeException: java.io.FileNotFoundException: File does not exist: /kylin/kylin_metadata/resources/GlobalDict/dict/DEFAULT.TEST_KYLIN_FACT/TEST_COUNT_DISTINCT_BITMAP/version_1496191754004/.index
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1828)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1799)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1712)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:652)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)

        at org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:129)
        at org.apache.kylin.cube.CubeManager.getDictionary(CubeManager.java:264)
        at org.apache.kylin.cube.CubeSegment.getDictionary(CubeSegment.java:329)
        at org.apache.kylin.cube.CubeSegment.buildDictionaryMap(CubeSegment.java:321)
        at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
        at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
        at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
        at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
        at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
        at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        at org.apache.spark.scheduler.Task.run(Task.scala:89)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

{code}

Then I checked the file system, there is only the ".index_v2" file:
{code}
[root@sandbox ~]# hadoop fs -ls /kylin/kylin_metadata/resources/GlobalDict/dict/DEFAULT.TEST_KYLIN_FACT/TEST_COUNT_DISTINCT_BITMAP/version_1496191754004/
Found 2 items
-rw-r--r--   3 shishaofeng hdfs         93 2017-05-31 08:49 /kylin/kylin_metadata/resources/GlobalDict/dict/DEFAULT.TEST_KYLIN_FACT/TEST_COUNT_DISTINCT_BITMAP/version_1496191754004/.index_v2
-rw-r--r--   3 shishaofeng hdfs       8036 2017-05-31 08:49 /kylin/kylin_metadata/resources/GlobalDict/dict/DEFAULT.TEST_KYLIN_FACT/TEST_COUNT_DISTINCT_BITMAP/version_1496191754004/cached_1496191753923_1
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)