Timeout visiting cube!

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Timeout visiting cube!

耳东
hi all

      When I query the cube, most of the sql can return results. However the more dimension included in the sql, the more times it takes to return the results. And when I add almost all the dimension in the sql , it returns Timeout visiting cube.






2016-05-20 10:12:41,492 ERROR [pool-12-thread-126] v2.CubeHBaseEndpointRPC:98 : Execution exception when running task in pool-12-thread-126
2016-05-20 10:12:41,492 ERROR [pool-12-thread-126] v2.CubeHBaseEndpointRPC:108 : Caught exception in thread pool-12-thread-126:
java.lang.RuntimeException: Error when visiting cubes by endpoint:
        at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1.run(CubeHBaseEndpointRPC.java:323)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=69040: row '' on table 'KYLIN_Z7VH6H8XII' at region=KYLIN_Z7VH6H8XII,,1463638140445.af341476d4df7a5e2ae9cf23a31afccf., hostname=test11,60020,1463636501632, seqNum=5
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
        at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:95)
        at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:56)
        at org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService$Stub.visitCube(CubeVisitProtos.java:3861)
        at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$2.call(CubeHBaseEndpointRPC.java:361)
        at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$2.call(CubeHBaseEndpointRPC.java:357)
        at org.apache.hadoop.hbase.client.HTable$16.call(HTable.java:1783)
        ... 4 more
Caused by: java.io.IOException: Call to test11/10.142.20.96:60020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=237083, waitTime=60001, operationTimeout=60000 expired.
        at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1236)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1204)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:300)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:31775)
        at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1618)
        at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:92)
        at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:89)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
        ... 10 more
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=237083, waitTime=60001, operationTimeout=60000 expired.
        at org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1178)
        ... 17 more
2016-05-20 10:12:41,492 ERROR [pool-12-thread-125] v2.CubeHBaseEndpointRPC:108 : Caught exception in thread pool-12-thread-125:
java.lang.RuntimeException: Error when visiting cubes by endpoint:
        at org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1.run(CubeHBaseEndpointRPC.java:323)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
Reply | Threaded
Open this post in threaded view
|

Re: Timeout visiting cube!

mahongbin
kylin query server can't wait for hbase coprosessors to return forever. By
default kylin uses hbase's "hbase.rpc.timeout" as the rpc call's timeout.
You can increase the kylin side timeout by increasing
the kylin.query.cube.visit.timeout.times to larger multiplier than
one(default). However it may still suffer from hbase rpc timeout

need to be 1.5.1 or above

On Fri, May 20, 2016 at 10:16 AM, 耳东 <[hidden email]> wrote:

> hi all
>
>       When I query the cube, most of the sql can return results. However
> the more dimension included in the sql, the more times it takes to return
> the results. And when I add almost all the dimension in the sql , it
> returns Timeout visiting cube.
>
>
>
>
>
>
> 2016-05-20 10:12:41,492 ERROR [pool-12-thread-126]
> v2.CubeHBaseEndpointRPC:98 : Execution exception when running task in
> pool-12-thread-126
> 2016-05-20 10:12:41,492 ERROR [pool-12-thread-126]
> v2.CubeHBaseEndpointRPC:108 : Caught exception in thread pool-12-thread-126:
> java.lang.RuntimeException: Error when visiting cubes by endpoint:
>         at
> org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1.run(CubeHBaseEndpointRPC.java:323)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:744)
> Caused by: java.net.SocketTimeoutException: callTimeout=60000,
> callDuration=69040: row '' on table 'KYLIN_Z7VH6H8XII' at
> region=KYLIN_Z7VH6H8XII,,1463638140445.af341476d4df7a5e2ae9cf23a31afccf.,
> hostname=test11,60020,1463636501632, seqNum=5
>         at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
>         at
> org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:95)
>         at
> org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:56)
>         at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService$Stub.visitCube(CubeVisitProtos.java:3861)
>         at
> org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$2.call(CubeHBaseEndpointRPC.java:361)
>         at
> org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$2.call(CubeHBaseEndpointRPC.java:357)
>         at org.apache.hadoop.hbase.client.HTable$16.call(HTable.java:1783)
>         ... 4 more
> Caused by: java.io.IOException: Call to test11/10.142.20.96:60020 failed
> on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call
> id=237083, waitTime=60001, operationTimeout=60000 expired.
>         at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1236)
>         at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1204)
>         at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216)
>         at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:300)
>         at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:31775)
>         at
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1618)
>         at
> org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:92)
>         at
> org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:89)
>         at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
>         ... 10 more
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call
> id=237083, waitTime=60001, operationTimeout=60000 expired.
>         at
> org.apache.hadoop.hbase.ipc.Call.checkAndSetTimeout(Call.java:70)
>         at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1178)
>         ... 17 more
> 2016-05-20 10:12:41,492 ERROR [pool-12-thread-125]
> v2.CubeHBaseEndpointRPC:108 : Caught exception in thread pool-12-thread-125:
> java.lang.RuntimeException: Error when visiting cubes by endpoint:
>         at
> org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1.run(CubeHBaseEndpointRPC.java:323)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:744)




--
Regards,

*Bin Mahone | 马洪宾*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone