When cube modified,can we query from all new and history segment without purge data?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

When cube modified,can we query from all new and history segment without purge data?

suheng.cloud
Hi, all
In reality, some small modify may be add into existing cube(like add a new measure),I search for some infomations that we could clone a new cube, but in this way, we must also rebuild all history data in new cube, or we can't query the sql in which part_date includes old and new values.

For example,kylin_sales_cube(partition start form 2012-01-01) build from 2012-01-01 to 2012-06-01
and
kylin_sales_cube_clone(partition start form 2012-09-01) build from 2012-09-01 to 2012-12-01

from now on,we want to stop incremental build for the formmer but change to build the latter one.(since more and more cube will hard to manager and take more compute and storage resource?)

sql:select part_dt, sum(price) as total_selled, count(distinct seller_id) as sellers from kylin_sales where part_dt > '2012-01-01'  group by part_dt order by part_dt;

only show up statistic from kylin_sales_cube

How can we make the sql show all data(merge result of 2012-01-01~2012-06-01 and 2012-09-01~2012-12-01) without rebuild all history segment?



Thanks a lot.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: When cube modified,can we query from all new and history segment without purge data?

shaofengshi
Check hybrid model: https://kylin.apache.org/blog/2015/09/25/hybrid-model/

2017-06-01 15:03 GMT+08:00 suheng.cloud <[hidden email]>:

> Hi, all
> In reality, some small modify may be add into existing cube(like add a new
> measure),I search for some infomations that we could clone a new cube, but
> in this way, we must also rebuild all history data in new cube, or we can't
> query the sql in which part_date includes old and new values.
>
> For example,kylin_sales_cube(partition start form 2012-01-01) build from
> 2012-01-01 to 2012-06-01
> and
> kylin_sales_cube_clone(partition start form 2012-09-01) build from
> 2012-09-01 to 2012-12-01
>
> from now on,we want to stop incremental build for the formmer but change to
> build the latter one.(since more and more cube will hard to manager and
> take
> more compute and storage resource?)
>
> sql:select part_dt, sum(price) as total_selled, count(distinct seller_id)
> as
> sellers from kylin_sales where part_dt > '2012-01-01'  group by part_dt
> order by part_dt;
>
> only show up statistic from kylin_sales_cube
>
> How can we make the sql show all data(merge result of 2012-01-01~2012-06-01
> and 2012-09-01~2012-12-01) without rebuild all history segment?
>
>
>
> Thanks a lot.
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/When-cube-modified-can-we-query-from-all-new-and-
> history-segment-without-purge-data-tp8138.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>



--
Best regards,

Shaofeng Shi 史少锋
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: When cube modified,can we query from all new and history segment without purge data?

suheng.cloud
Thanks, shaofeng ,it works.
This artical is the only one I missed..
It would be nice if integrated into web or restful service in futhure release :)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: When cube modified,can we query from all new and history segment without purge data?

shaofengshi
Suheng, thanks for your suggestion, yes they should be linked in the
documentation, we will do that.

2017-06-01 15:48 GMT+08:00 suheng.cloud <[hidden email]>:

> Thanks, shaofeng ,it works.
> This artical is the only one I missed..
> It would be nice if integrated into web or restful service in futhure
> release :)
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/When-cube-modified-can-we-query-from-all-new-and-
> history-segment-without-purge-data-tp8138p8146.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>



--
Best regards,

Shaofeng Shi 史少锋
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: When cube modified,can we query from all new and history segment without purge data?

Billy Liu
https://issues.apache.org/jira/browse/KYLIN-2317 provides CLI tools

2017-06-01 15:59 GMT+08:00 ShaoFeng Shi <[hidden email]>:

> Suheng, thanks for your suggestion, yes they should be linked in the
> documentation, we will do that.
>
> 2017-06-01 15:48 GMT+08:00 suheng.cloud <[hidden email]>:
>
> > Thanks, shaofeng ,it works.
> > This artical is the only one I missed..
> > It would be nice if integrated into web or restful service in futhure
> > release :)
> >
> > --
> > View this message in context: http://apache-kylin.74782.x6.
> > nabble.com/When-cube-modified-can-we-query-from-all-new-and-
> > history-segment-without-purge-data-tp8138p8146.html
> > Sent from the Apache Kylin mailing list archive at Nabble.com.
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
Loading...