kylin2.0.0版本 streaming build 一直在pending 状态

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

kylin2.0.0版本 streaming build 一直在pending 状态

88265920
想尝试kylin 的 streaming 构建,但是build的时候一直在pending状态,下面是任务信息

Job Name test - 50_55 - BUILD - GMT+08:00 2017-06-09 14:57:32
Job ID 67cf1818-9ac2-4303-91b1-e21d4125fba1
Status PENDING
Duration 0.00 mins
MapReduce Waiting 0.00 mins


#1 Step Name: Save data from Kafka
Duration: 0 seconds Waiting: 0 seconds
#2 Step Name: Extract Fact Table Distinct Columns
Duration: 0 seconds Waiting: 0 seconds
#3 Step Name: Build Dimension Dictionary
Duration: 0 seconds Waiting: 0 seconds
#4 Step Name: Save Cuboid Statistics
Duration: 0 seconds Waiting: 0 seconds
#5 Step Name: Create HTable
Duration: 0 seconds Waiting: 0 seconds
#6 Step Name: Build Cube with Spark
Duration: 0 seconds Waiting: 0 seconds
#7 Step Name: Convert Cuboid Data to HFile
Duration: 0 seconds Waiting: 0 seconds
#8 Step Name: Load HFile to HBase Table
Duration: 0 seconds Waiting: 0 seconds
#9 Step Name: Update Cube Info
Duration: 0 seconds Waiting: 0 seconds
#10 Step Name: Kafka Intermediate File Cleanup
Duration: 0 seconds Waiting: 0 seconds

请问这个是什么原因呢?
Reply | Threaded
Open this post in threaded view
|

Re: kylin2.0.0版本 streaming build 一直在pending 状态

88265920
public class ZookeeperJobLock implements JobLock {
    private Logger logger = LoggerFactory.getLogger(ZookeeperJobLock.class);

    private static final String ZOOKEEPER_LOCK_PATH = "/kylin/job_engine/lock";

    // 一直停留在这行代码
    hasLock = sharedLock.acquire(3, TimeUnit.SECONDS);

问题原因是 zookeeper创建分布式锁的时候,系统目录不存在,代码停留在锁的那一行了,导致整个任务调度线程没有启动

解决:
sudo mkdir /kylin/
sudo mkdir /kylin/job_engine/
sudo mkdir /kylin/job_engine/lock
sudo chown kangjiao:kangjiao /kylin/job_engine/lock/
Reply | Threaded
Open this post in threaded view
|

Re: kylin2.0.0版本 streaming build 一直在pending 状态

shaofengshi
Hi, Kylin uses a ZK lock to ensure only one Job engine instance is running.
The lock path "/kylin/job_engine/lock" is a path on Zookeeper, not a local
or HDFS folder I think. Can you double confirm your probem was solved by
making a local dir? Thanks.

2017-06-10 16:06 GMT+08:00 88265920 <[hidden email]>:

> public class ZookeeperJobLock implements JobLock {
>     private Logger logger = LoggerFactory.getLogger(
> ZookeeperJobLock.class);
>
>     private static final String ZOOKEEPER_LOCK_PATH =
> "/kylin/job_engine/lock";
>
>     // 一直停留在这行代码
>     hasLock = sharedLock.acquire(3, TimeUnit.SECONDS);
>
> 问题原因是 zookeeper创建分布式锁的时候,系统目录不存在,代码停留在锁的那一行了,导致整个任务调度线程没有启动
>
> 解决:
> sudo mkdir /kylin/
> sudo mkdir /kylin/job_engine/
> sudo mkdir /kylin/job_engine/lock
> sudo chown kangjiao:kangjiao /kylin/job_engine/lock/
>
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/kylin2-0-0-streaming-build-pending-tp8202p8206.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>



--
Best regards,

Shaofeng Shi 史少锋