Timeout when refreshing a cube

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Timeout when refreshing a cube

bk_180
This post was updated on .
Hi all,

We are facing some errors due to timeout exception.

{"url":"http://<masternode>:7070/kylin/api/jobs/349d62e3-7748-46ee-8009-3085726b5933","exception":"java.lang.RuntimeException: org.apache.kylin.job.exception.PersistentException: java.net.SocketTimeoutException: callTimeout=60000, callDuration=63012: row '/execute/349d62e3-7748-46ee-8009-3085726b5933' on table 'kylin_metadata' at region=kylin_metadata,/dict/VIEW_CREDITO/COD_PROMOTOR/a61c83e7-01e7-4971-9539-dd5c7ed07dff.dict,1500376865680.301b695304237ff0d1a7774bbff83b67., hostname=<node>,16020,1501960877256, seqNum=163046"}----------------------------------------------------------------------

We have already changed the following properties hbase.client.scanner.timeout.period and hbase.rpc.timeout to 600.000.  Is there any other property to change?

Sometimes, after getting timeout error, some jobs just disappear from web monitor or get its status changed to Error.

We are also facing some errors like "java.lang.IllegalStateException: No resource found at -- /dict/", is there any chance this occur because of timeout error?

Kylin_log_timeout.Kylin_log_timeout
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Timeout when refreshing a cube

shaofengshi
I ever faced this error in one of my environments. That env has several
streaming cubes building every couple mintues, so lots of files accumulated
in the metadata table. Run a metadata cleanup solved that problem (follow
the cleanup step in
https://kylin.apache.org/docs20/howto/howto_backup_metadata.html).

You also need check HBase's status to see whether there is offline regions.

2017-08-07 23:29 GMT+08:00 bk_180 <[hidden email]>:

> Hi all,
>
> We are facing some errors due to timeout exception.
>
> {"url":"http://<masternode>:7070/kylin/api/jobs/349d62e3-
> 7748-46ee-8009-3085726b5933","exception":"java.lang.RuntimeException:
> org.apache.kylin.job.exception.PersistentException:
> java.net.SocketTimeoutException: callTimeout=60000, callDuration=63012:
> row
> '/execute/349d62e3-7748-46ee-8009-3085726b5933' on table 'kylin_metadata'
> at
> region=kylin_metadata,/dict/VIEW_CREDITO/COD_PROMOTOR/
> a61c83e7-01e7-4971-9539-dd5c7ed07dff.dict,1500376865680.
> 301b695304237ff0d1a7774bbff83b67.,
> hostname=<node>,16020,1501960877256,
> seqNum=163046"}---------------------------------------------
> -------------------------
>
> We have already changed the following properties
> hbase.client.scanner.timeout.period and hbase.rpc.timeout to 600.000.  Is
> there any other property to change?
>
> Sometimes, after getting timeout error Kylin_log_timeout.Kylin_log_timeout
> <http://apache-kylin.74782.x6.nabble.com/file/n8658/Kylin_
> log_timeout.Kylin_log_timeout>
> s, some jobs just disappear from web monitor or get its status changed to
> Error.
>
> We are also facing some errors like "java.lang.IllegalStateException: No
> resource found at -- /dict/", is there any chance this occur because of
> timeout error?
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/Timeout-when-refreshing-a-cube-tp8658.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>



--
Best regards,

Shaofeng Shi 史少锋
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Timeout when refreshing a cube

bk_180
Hi Shaofeng Shi,


Looking for some information on Hbase UI I realized that the node informed in the log was stopped during the process. So this explaind the timeout, correct?

{"url":"http://<masternode>:7070/kylin/api/jobs/349d62e3-7748-46ee-8009-3085726b5933","exception":"java.lang.RuntimeException: org.apache.kylin.job.exception.PersistentException: java.net.SocketTimeoutException: callTimeout=60000, callDuration=63012: row '/execute/349d62e3-7748-46ee-8009-3085726b5933' on table 'kylin_metadata' at region=kylin_metadata,/dict/VIEW_CREDITO/COD_PROMOTOR/a61c83e7-01e7-4971-9539-dd5c7ed07dff.dict,1500376865680.301b695304237ff0d1a7774bbff83b67., hostname=<node>,16020,1501960877256, seqNum=163046"}

We are running a Cluster on AWS EMR (hbase in S3) and using the tasknodes with autoscalling feature turned on, I´m understandig that this node was decomissioned due this configuration.

Can you see any other problem we migth face?    

Thanks for answering my post.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Timeout when refreshing a cube

shaofengshi
Kylin relies on HBase not only for Cube, but also for meta data. A stable
HBase is the basis if you expect Kylin with high performance and high
availability.  I don't know how EMR can ensure HBase HA when auto-scaling;
if it couldn't, that would be a problem.


2017-08-09 0:19 GMT+08:00 bk_180 <[hidden email]>:

> Hi Shaofeng Shi,
>
>
> Looking for some information on Hbase UI I realized that the node informed
> in the log was stopped during the process. So this explaind the timeout,
> correct?
>
> {"url":"http://<masternode>:7070/kylin/api/jobs/349d62e3-
> 7748-46ee-8009-3085726b5933","exception":"java.lang.RuntimeException:
> org.apache.kylin.job.exception.PersistentException:
> java.net.SocketTimeoutException: callTimeout=60000, callDuration=63012:
> row
> '/execute/349d62e3-7748-46ee-8009-3085726b5933' on table 'kylin_metadata'
> at
> region=kylin_metadata,/dict/VIEW_CREDITO/COD_PROMOTOR/
> a61c83e7-01e7-4971-9539-dd5c7ed07dff.dict,1500376865680.
> 301b695304237ff0d1a7774bbff83b67.,
> /hostname=<node>,/16020,1501960877256, seqNum=163046"}
>
> We are running a Cluster on AWS EMR (hbase in S3) and using the tasknodes
> with autoscalling feature turned on, I´m understandig that this node was
> decomissioned due this configuration.
>
> Can you see any other problem we migth face?
>
> Thanks for answering my post.
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/Timeout-when-refreshing-a-cube-tp8658p8671.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>



--
Best regards,

Shaofeng Shi 史少锋
Loading...