Kylin go to hdfs to find jar file

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Kylin go to hdfs to find jar file

shaonian
hello,here is my question:
I try to use kylin, here is my install enviroment:


apache hadoop 2.7.3
apache hbase 1.3.0
apache hive 2.1.1
apache kylin 2.0.0


I followed quick start sample to do,but when I build a cube,there is a error occured in Step 3:
java.io.FileNotFoundException: File does not exist: hdfs://localhost:9000/opt/bigdata/hive-2.1.1/lib/hive-exec-2.1.1.jar
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072)
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
        at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
        at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)
        at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:99)
        at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)
        at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
        at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
        at org.apache.kylin.engine.mr.common.AbstractHadoopJob.waitForCompletion(AbstractHadoopJob.java:149)
        at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:130)
        at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:102)
        at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:123)
        at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
        at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
        at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
        at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:142)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
I don't know why kylin would go to hdfs to find jar file.




best wishes
Chai shiyu
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Fwd: Kylin go to hdfs to find jar file

Luke Han
Administrator
Forward to dev mailing list

Best Regards!
---------------------

Luke Han

---------- Forwarded message ----------
From: 柴诗雨 <[hidden email]>
Date: Wed, May 31, 2017 at 9:58 PM
Subject: Kylin go to hdfs to find jar file
To: [hidden email]


hello,here is my question:
I try to use kylin, here is my install enviroment:


apache hadoop 2.7.3
apache hbase 1.3.0
apache hive 2.1.1
apache kylin 2.0.0


I followed quick start sample to do,but when I build a cube,there is a
error occured in Step 3:
java.io.FileNotFoundException: File does not exist:
hdfs://localhost:9000/opt/bigdata/hive-2.1.1/lib/hive-exec-2.1.1.jar
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.
doCall(DistributedFileSystem.java:1072)
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.
doCall(DistributedFileSystem.java:1064)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(
FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(
DistributedFileSystem.java:1064)
        at org.apache.hadoop.mapreduce.filecache.
ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.
java:288)
        at org.apache.hadoop.mapreduce.filecache.
ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.
java:224)
        at org.apache.hadoop.mapreduce.filecache.
ClientDistributedCacheManager.determineTimestamps(
ClientDistributedCacheManager.java:99)
        at org.apache.hadoop.mapreduce.filecache.
ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(
ClientDistributedCacheManager.java:57)
        at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(
JobSubmitter.java:265)
        at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(
JobSubmitter.java:301)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(
JobSubmitter.java:389)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(
UserGroupInformation.java:1614)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
        at org.apache.kylin.engine.mr.common.AbstractHadoopJob.
waitForCompletion(AbstractHadoopJob.java:149)
        at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
run(FactDistinctColumnsJob.java:130)
        at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:102)
        at org.apache.kylin.engine.mr.common.MapReduceExecutable.
doWork(MapReduceExecutable.java:123)
        at org.apache.kylin.job.execution.AbstractExecutable.
execute(AbstractExecutable.java:124)
        at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
DefaultChainedExecutable.java:64)
        at org.apache.kylin.job.execution.AbstractExecutable.
execute(AbstractExecutable.java:124)
        at org.apache.kylin.job.impl.threadpool.DefaultScheduler$
JobRunner.run(DefaultScheduler.java:142)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
I don't know why kylin would go to hdfs to find jar file.




best wishes
Chai shiyu
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Kylin go to hdfs to find jar file

Billy Liu
In reply to this post by shaonian
Please check the find-hive-dependency.sh, make sure the hive_dependency
works as expected.

2017-05-31 21:58 GMT+08:00 柴诗雨 <[hidden email]>:

> hello,here is my question:
> I try to use kylin, here is my install enviroment:
>
>
> apache hadoop 2.7.3
> apache hbase 1.3.0
> apache hive 2.1.1
> apache kylin 2.0.0
>
>
> I followed quick start sample to do,but when I build a cube,there is a
> error occured in Step 3:
> java.io.FileNotFoundException: File does not exist:
> hdfs://localhost:9000/opt/bigdata/hive-2.1.1/lib/hive-exec-2.1.1.jar
>         at org.apache.hadoop.hdfs.DistributedFileSystem$17.
> doCall(DistributedFileSystem.java:1072)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$17.
> doCall(DistributedFileSystem.java:1064)
>         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(
> FileSystemLinkResolver.java:81)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(
> DistributedFileSystem.java:1064)
>         at org.apache.hadoop.mapreduce.filecache.
> ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.
> java:288)
>         at org.apache.hadoop.mapreduce.filecache.
> ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.
> java:224)
>         at org.apache.hadoop.mapreduce.filecache.
> ClientDistributedCacheManager.determineTimestamps(
> ClientDistributedCacheManager.java:99)
>         at org.apache.hadoop.mapreduce.filecache.
> ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(
> ClientDistributedCacheManager.java:57)
>         at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(
> JobSubmitter.java:265)
>         at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(
> JobSubmitter.java:301)
>         at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(
> JobSubmitter.java:389)
>         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
>         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1614)
>         at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
>         at org.apache.kylin.engine.mr.common.AbstractHadoopJob.
> waitForCompletion(AbstractHadoopJob.java:149)
>         at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> run(FactDistinctColumnsJob.java:130)
>         at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:102)
>         at org.apache.kylin.engine.mr.common.MapReduceExecutable.
> doWork(MapReduceExecutable.java:123)
>         at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:124)
>         at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:64)
>         at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:124)
>         at org.apache.kylin.job.impl.threadpool.DefaultScheduler$
> JobRunner.run(DefaultScheduler.java:142)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> I don't know why kylin would go to hdfs to find jar file.
>
>
>
>
> best wishes
> Chai shiyu
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Kylin go to hdfs to find jar file

KevinLei
Hi Billy,

I have the similar issue. Could you please help on this?

Here is the output error info:

java.io.FileNotFoundException: File does not exist: hdfs://hkpcva01:54310/hadoop/hive/lib/accumulo-core-1.6.0.jar

Here is the output of find-hive-dependency.sh

/hadoop/hive/conf:/hadoop/hive/lib/hive-service-1.2.2.jar:/hadoop/hive/lib/accumulo-trace-1.6.0.jar:/hadoop/hive/lib/curator-framework-2.6.0.jar:/hadoop/hive/lib/maven-scm-provider-svn-commons-1.4.jar:/hadoop/hive/lib/commons-logging-1.1.3.jar:/hadoop/hive/lib/jline-2.12.jar:/hadoop/hive/lib/asm-commons-3.1.jar:/hadoop/hive/lib/apache-log4j-extras-1.2.17.jar:/hadoop/hive/lib/commons-cli-1.2.jar:/hadoop/hive/lib/hive-testutils-1.2.2.jar:/hadoop/hive/lib/hive-shims-1.2.2.jar:/hadoop/hive/lib/commons-beanutils-core-1.8.0.jar:/hadoop/hive/lib/curator-client-2.6.0.jar:/hadoop/hive/lib/regexp-1.3.jar:/hadoop/hive/lib/commons-configuration-1.6.jar:/hadoop/hive/lib/hive-shims-common-1.2.2.jar:/hadoop/hive/lib/datanucleus-rdbms-3.2.9.jar:/hadoop/hive/lib/accumulo-start-1.6.0.jar:/hadoop/hive/lib/hive-exec-1.2.2.jar:/hadoop/hive/lib/guava-14.0.1.jar:/hadoop/hive/lib/super-csv-2.2.0.jar:/hadoop/hive/lib/xz-1.0.jar:/hadoop/hive/lib/commons-pool-1.5.4.jar:/hadoop/hive/lib/accumulo-fate-1.6.0.jar:/hadoop/hive/lib/hive-accumulo-handler-1.2.2.jar:/hadoop/hive/lib/hive-contrib-1.2.2.jar:/hadoop/hive/lib/commons-compress-1.4.1.jar:/hadoop/hive/lib/ant-launcher-1.9.1.jar:/hadoop/hive/lib/opencsv-2.3.jar:/hadoop/hive/lib/paranamer-2.3.jar:/hadoop/hive/lib/mail-1.4.1.jar:/hadoop/hive/lib/geronimo-jta_1.1_spec-1.1.1.jar:/hadoop/hive/lib/commons-io-2.4.jar:/hadoop/hive/lib/datanucleus-core-3.2.10.jar:/hadoop/hive/lib/commons-dbcp-1.4.jar:/hadoop/hive/lib/zookeeper-3.4.6.jar:/hadoop/hive/lib/tempus-fugit-1.1.jar:/hadoop/hive/lib/curator-recipes-2.6.0.jar:/hadoop/hive/lib/ant-1.9.1.jar:/hadoop/hive/lib/log4j-1.2.16.jar:/hadoop/hive/lib/stax-api-1.0.1.jar:/hadoop/hive/lib/joda-time-2.5.jar:/hadoop/hive/lib/jdo-api-3.0.1.jar:/hadoop/hive/lib/hive-hwi-1.2.2.jar:/hadoop/hive/lib/commons-beanutils-1.7.0.jar:/hadoop/hive/lib/hive-beeline-1.2.2.jar:/hadoop/hive/lib/commons-compiler-2.7.6.jar:/hadoop/hive/lib/hive-common-1.2.2.jar:/hadoop/hive/lib/avro-1.7.5.jar:/hadoop/hive/lib/httpclient-4.4.jar:/hadoop/hive/lib/commons-math-2.1.jar:/hadoop/hive/lib/hive-jdbc-1.2.2.jar:/hadoop/hive/lib/activation-1.1.jar:/hadoop/hive/lib/hive-hbase-handler-1.2.2.jar:/hadoop/hive/lib/commons-vfs2-2.0.jar:/hadoop/hive/lib/groovy-all-2.1.6.jar:/hadoop/hive/lib/commons-codec-1.4.jar:/hadoop/hive/lib/junit-4.11.jar:/hadoop/hive/lib/commons-httpclient-3.0.1.jar:/hadoop/hive/lib/janino-2.7.6.jar:/hadoop/hive/lib/jcommander-1.32.jar:/hadoop/hive/lib/jta-1.1.jar:/hadoop/hive/lib/hive-ant-1.2.2.jar:/hadoop/hive/lib/bonecp-0.8.0.RELEASE.jar:/hadoop/hive/lib/plexus-utils-1.5.6.jar:/hadoop/hive/lib/commons-collections-3.2.2.jar:/hadoop/hive/lib/asm-tree-3.1.jar:/hadoop/hive/lib/eigenbase-properties-1.1.5.jar:/hadoop/hive/lib/libfb303-0.9.2.jar:/hadoop/hive/lib/ST4-4.0.4.jar:/hadoop/hive/lib/hamcrest-core-1.1.jar:/hadoop/hive/lib/commons-lang-2.6.jar:/hadoop/hive/lib/antlr-2.7.7.jar:/hadoop/hive/lib/netty-3.7.0.Final.jar:/hadoop/hive/lib/hive-serde-1.2.2.jar:/hadoop/hive/lib/snappy-java-1.0.5.jar:/hadoop/hive/lib/jsr305-3.0.0.jar:/hadoop/hive/lib/maven-scm-provider-svnexe-1.4.jar:/hadoop/hive/lib/datanucleus-api-jdo-3.2.6.jar:/hadoop/hive/lib/json-20090211.jar:/hadoop/hive/lib/servlet-api-2.5.jar:/hadoop/hive/lib/velocity-1.5.jar:/hadoop/hive/lib/oro-2.0.8.jar:/hadoop/hive/lib/hive-shims-0.23-1.2.2.jar:/hadoop/hive/lib/jetty-all-server-7.6.0.v20120127.jar:/hadoop/hive/lib/httpcore-4.4.jar:/hadoop/hive/lib/mysql-connector-java-5.1.42-bin.jar:/hadoop/hive/lib/jpam-1.1.jar:/hadoop/hive/lib/commons-digester-1.8.jar:/hadoop/hive/lib/hive-metastore-1.2.2.jar:/hadoop/hive/lib/stringtemplate-3.2.1.jar:/hadoop/hive/lib/geronimo-annotation_1.0_spec-1.1.1.jar:/hadoop/hive/lib/hive-jdbc-1.2.2-standalone.jar:/hadoop/hive/lib/hive-cli-1.2.2.jar:/hadoop/hive/lib/derby-10.10.2.0.jar:/hadoop/hive/lib/antlr-runtime-3.4.jar:/hadoop/hive/lib/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar:/hadoop/hive/lib/geronimo-jaspic_1.0_spec-1.0.jar:/hadoop/hive/lib/accumulo-core-1.6.0.jar:/hadoop/hive/lib/libthrift-0.9.2.jar:/hadoop/hive/lib/maven-scm-api-1.4.jar:/hadoop/hive/lib/ivy-2.4.0.jar:/hadoop/hive/lib/parquet-hadoop-bundle-1.6.0.jar:/hadoop/hive/lib/jetty-all-7.6.0.v20120127.jar:/hadoop/hive/lib/hive-shims-scheduler-1.2.2.jar:/hadoop/hive/lib/hive-shims-0.20S-1.2.2.jar:/hadoop/hive/hcatalog/share/hcatalog/hive-hcatalog-core-1.2.2.jar
Loading...