Hbase on S3

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Hbase on S3

Tokayer, Jason M.
Does Kylin support storing data in S3 rather than HDFS?

________________________________________________________

The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Reply | Threaded
Open this post in threaded view
|

Re: Hbase on S3

shaofengshi
Hi Tokayer,

Yes it works; I just verified that works, the only additional work is
copying one property from /etc/hbase/conf.dist/hbase-site.xml to
KYLIN_HOME/conf/kylin_job_conf.xml:

  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>ip-10-0-0-222.us-west-2.compute.internal</value>
  </property>

Otherwise the "Convert to HFile" step will got zk connection error.

After built, the data will be loaded to S3 for HBase, e.g:

hbase(main):002:0> describe 'KYLIN_ROXUG026FC'
Table KYLIN_ROXUG026FC is ENABLED
KYLIN_ROXUG026FC, {TABLE_ATTRIBUTES => {coprocessor$1 =>
'hdfs://ip-10-0-0-222.us-west-2.compute.internal:8020/kylin/kylin_metadata/coprocessor/
kylin-coprocessor-2.0.0-0.jar|org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService|1001|',
METADATA => {'CREATION_TIME'
=> '1496192721720', 'GIT_COMMIT' =>
'375fd807c281d8c5deff0620747c806be2019782;', 'KYLIN_HOST' =>
'kylin_metadata', 'OWNER' => '[hidden email]
he.org', 'SEGMENT' => 'kylin_sales_cube[20120101000000_20170501000000]',
'SPLIT_POLICY' => 'org.apache.hadoop.hbase.regionserver.DisabledRegionS
plitPolicy'}}


[hadoop@ip-10-0-0-222 apache-kylin-2.0.0-bin]$ hadoop fs -ls
s3://kap-test-hbase-buket1/data/default/KYLIN_ROXUG026FC
Found 6 items
drwxrwxrwx   - hadoop hadoop          0 1970-01-01 00:00
s3://kap-test-hbase-buket1/data/default/KYLIN_ROXUG026FC/.tabledesc
drwxrwxrwx   - hadoop hadoop          0 1970-01-01 00:00
s3://kap-test-hbase-buket1/data/default/KYLIN_ROXUG026FC/12b468c028a2be8f6c781d6acf28d36b
drwxrwxrwx   - hadoop hadoop          0 1970-01-01 00:00
s3://kap-test-hbase-buket1/data/default/KYLIN_ROXUG026FC/4fd1f27663a7216f8de833224d099ad4
drwxrwxrwx   - hadoop hadoop          0 1970-01-01 00:00
s3://kap-test-hbase-buket1/data/default/KYLIN_ROXUG026FC/b655b7e35df6908d204bb3854e9eef0b
drwxrwxrwx   - hadoop hadoop          0 1970-01-01 00:00
s3://kap-test-hbase-buket1/data/default/KYLIN_ROXUG026FC/e6ab10cd14cb5830eb80cda78dff7a61
drwxrwxrwx   - hadoop hadoop          0 1970-01-01 00:00
s3://kap-test-hbase-buket1/data/default/KYLIN_ROXUG026FC/f31928db11f79dbd8961880d099f6e2b




2017-05-30 21:37 GMT+08:00 Tokayer, Jason M. <[hidden email]>:

> Does Kylin support storing data in S3 rather than HDFS?
>
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>



--
Best regards,

Shaofeng Shi 史少锋
Reply | Threaded
Open this post in threaded view
|

Re: Hbase on S3

shaofengshi
Don't forget to use the IP address with your environment's.

2017-05-31 9:42 GMT+08:00 ShaoFeng Shi <[hidden email]>:

> Hi Tokayer,
>
> Yes it works; I just verified that works, the only additional work is
> copying one property from /etc/hbase/conf.dist/hbase-site.xml to
> KYLIN_HOME/conf/kylin_job_conf.xml:
>
>   <property>
>     <name>hbase.zookeeper.quorum</name>
>     <value>ip-10-0-0-222.us-west-2.compute.internal</value>
>   </property>
>
> Otherwise the "Convert to HFile" step will got zk connection error.
>
> After built, the data will be loaded to S3 for HBase, e.g:
>
> hbase(main):002:0> describe 'KYLIN_ROXUG026FC'
> Table KYLIN_ROXUG026FC is ENABLED
> KYLIN_ROXUG026FC, {TABLE_ATTRIBUTES => {coprocessor$1 =>
> 'hdfs://ip-10-0-0-222.us-west-2.compute.internal:8020/kylin/
> kylin_metadata/coprocessor/
> kylin-coprocessor-2.0.0-0.jar|org.apache.kylin.storage.
> hbase.cube.v2.coprocessor.endpoint.CubeVisitService|1001|', METADATA =>
> {'CREATION_TIME'
> => '1496192721720', 'GIT_COMMIT' => '375fd807c281d8c5deff0620747c806be2019782;',
> 'KYLIN_HOST' => 'kylin_metadata', 'OWNER' => '[hidden email]
> he.org', 'SEGMENT' => 'kylin_sales_cube[20120101000000_20170501000000]',
> 'SPLIT_POLICY' => 'org.apache.hadoop.hbase.regionserver.DisabledRegionS
> plitPolicy'}}
>
>
> [hadoop@ip-10-0-0-222 apache-kylin-2.0.0-bin]$ hadoop fs -ls
> s3://kap-test-hbase-buket1/data/default/KYLIN_ROXUG026FC
> Found 6 items
> drwxrwxrwx   - hadoop hadoop          0 1970-01-01 00:00
> s3://kap-test-hbase-buket1/data/default/KYLIN_ROXUG026FC/.tabledesc
> drwxrwxrwx   - hadoop hadoop          0 1970-01-01 00:00
> s3://kap-test-hbase-buket1/data/default/KYLIN_ROXUG026FC/
> 12b468c028a2be8f6c781d6acf28d36b
> drwxrwxrwx   - hadoop hadoop          0 1970-01-01 00:00
> s3://kap-test-hbase-buket1/data/default/KYLIN_ROXUG026FC/
> 4fd1f27663a7216f8de833224d099ad4
> drwxrwxrwx   - hadoop hadoop          0 1970-01-01 00:00
> s3://kap-test-hbase-buket1/data/default/KYLIN_ROXUG026FC/
> b655b7e35df6908d204bb3854e9eef0b
> drwxrwxrwx   - hadoop hadoop          0 1970-01-01 00:00
> s3://kap-test-hbase-buket1/data/default/KYLIN_ROXUG026FC/
> e6ab10cd14cb5830eb80cda78dff7a61
> drwxrwxrwx   - hadoop hadoop          0 1970-01-01 00:00
> s3://kap-test-hbase-buket1/data/default/KYLIN_ROXUG026FC/
> f31928db11f79dbd8961880d099f6e2b
>
>
>
>
> 2017-05-30 21:37 GMT+08:00 Tokayer, Jason M. <[hidden email]
> >:
>
>> Does Kylin support storing data in S3 rather than HDFS?
>>
>> ________________________________________________________
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The information
>> transmitted herewith is intended only for use by the individual or entity
>> to which it is addressed. If the reader of this message is not the intended
>> recipient, you are hereby notified that any review, retransmission,
>> dissemination, distribution, copying or other use of, or taking of any
>> action in reliance upon this information is strictly prohibited. If you
>> have received this communication in error, please contact the sender and
>> delete the material from your computer.
>>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>


--
Best regards,

Shaofeng Shi 史少锋