How to migrate cube from one Kylin to another (for example, Kylin QA to Kylin Production) within the different Hadoop Cluster

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How to migrate cube from one Kylin to another (for example, Kylin QA to Kylin Production) within the different Hadoop Cluster

wangzy24
What would you do if you wanted to do metadata migration between completely independent environments within the different Hadoop Cluster?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to migrate cube from one Kylin to another (for example, Kylin QA to Kylin Production) within the different Hadoop Cluster

shaofengshi
Kylin metadata migration is very simple, just follow the "backup/restore
metadata" guide.

But if you want to have the Cube data also be moved to new cluster, you
need migrate:

1) Hive tables (metadata and files);
2) Files in Kylin HDFS working dir;
3) HBase tables (KYLIN_.*)

There is no a tool to automate these steps. You need do that by your own.

2017-08-03 15:06 GMT+08:00 wangzy24 <[hidden email]>:

> What would you do if you wanted to do metadata migration between completely
> independent environments within the different Hadoop Cluster?
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/How-to-migrate-cube-from-one-Kylin-to-
> another-for-example-Kylin-QA-to-Kylin-Production-within-
> the-dir-tp8628.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>



--
Best regards,

Shaofeng Shi 史少锋
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to migrate cube from one Kylin to another (for example, Kylin QA to Kylin Production) within the different Hadoop Cluster

Yang
Rebuild the cube in new cluster is often easier.

On Thu, Aug 3, 2017 at 8:29 PM, ShaoFeng Shi <[hidden email]> wrote:

> Kylin metadata migration is very simple, just follow the "backup/restore
> metadata" guide.
>
> But if you want to have the Cube data also be moved to new cluster, you
> need migrate:
>
> 1) Hive tables (metadata and files);
> 2) Files in Kylin HDFS working dir;
> 3) HBase tables (KYLIN_.*)
>
> There is no a tool to automate these steps. You need do that by your own.
>
> 2017-08-03 15:06 GMT+08:00 wangzy24 <[hidden email]>:
>
> > What would you do if you wanted to do metadata migration between
> completely
> > independent environments within the different Hadoop Cluster?
> >
> > --
> > View this message in context: http://apache-kylin.74782.x6.
> > nabble.com/How-to-migrate-cube-from-one-Kylin-to-
> > another-for-example-Kylin-QA-to-Kylin-Production-within-
> > the-dir-tp8628.html
> > Sent from the Apache Kylin mailing list archive at Nabble.com.
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to migrate cube from one Kylin to another (for example, Kylin QA to Kylin Production) within the different Hadoop Cluster

wangzy24
In reply to this post by shaofengshi
Not to do the whole data migration, it is iterative development, the demand upgrade, the part of the metadata that we want to migrate, what could be the solution?


shaofengshi wrote
Kylin metadata migration is very simple, just follow the "backup/restore
metadata" guide.

But if you want to have the Cube data also be moved to new cluster, you
need migrate:

1) Hive tables (metadata and files);
2) Files in Kylin HDFS working dir;
3) HBase tables (KYLIN_.*)

There is no a tool to automate these steps. You need do that by your own.

2017-08-03 15:06 GMT+08:00 wangzy24 <[hidden email]>:

> What would you do if you wanted to do metadata migration between completely
> independent environments within the different Hadoop Cluster?
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/How-to-migrate-cube-from-one-Kylin-to-
> another-for-example-Kylin-QA-to-Kylin-Production-within-
> the-dir-tp8628.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>



--
Best regards,

Shaofeng Shi 史少锋
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to migrate cube from one Kylin to another (for example, Kylin QA to Kylin Production) within the different Hadoop Cluster

Yang
Then pls following Shaofeng's answer.

On Tue, Aug 8, 2017 at 4:51 PM, wangzy24 <[hidden email]> wrote:

> Not to do the whole data migration, it is iterative development, the demand
> upgrade, the part of the metadata that we want to migrate, what could be
> the
> solution?
>
>
>
> shaofengshi wrote
> > Kylin metadata migration is very simple, just follow the "backup/restore
> > metadata" guide.
> >
> > But if you want to have the Cube data also be moved to new cluster, you
> > need migrate:
> >
> > 1) Hive tables (metadata and files);
> > 2) Files in Kylin HDFS working dir;
> > 3) HBase tables (KYLIN_.*)
> >
> > There is no a tool to automate these steps. You need do that by your own.
> >
> > 2017-08-03 15:06 GMT+08:00 wangzy24 &lt;
>
> > wangzy24@
>
> > &gt;:
> >
> >> What would you do if you wanted to do metadata migration between
> >> completely
> >> independent environments within the different Hadoop Cluster?
> >>
> >> --
> >> View this message in context: http://apache-kylin.74782.x6.
> >> nabble.com/How-to-migrate-cube-from-one-Kylin-to-
> >> another-for-example-Kylin-QA-to-Kylin-Production-within-
> >> the-dir-tp8628.html
> >> Sent from the Apache Kylin mailing list archive at Nabble.com.
> >>
> >
> >
> >
> > --
> > Best regards,
> >
> > Shaofeng Shi 史少锋
>
>
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/How-to-migrate-cube-from-one-Kylin-to-
> another-for-example-Kylin-QA-to-Kylin-Production-within-
> the-dir-tp8628p8666.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>
Loading...