Hbase错误解析： Call queue is full on /0.0.0.0:60020, too many items queued ?
完整错误日志如下:
2020-02-15 09:21:19,659 INFO org.apache.hadoop.hbase.client.AsyncProcess - , tableName=bd_push_device
2020-02-15 09:21:39,795 INFO org.apache.hadoop.hbase.client.AsyncProcess - #21692403, table=bd_push_device, attempt=28/35 failed=2ops, last exception: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException): Call queue is full on /0.0.0.0:60020, too many items queued ? on lvzhuweng,60020,1581302282313, tracking started null, retrying after=20093ms, replay=2ops
2020-02-15 09:21:39,801 INFO org.apache.hadoop.hbase.client.AsyncProcess - , tableName=bd_push_device
2020-02-15 09:21:59,891 INFO org.apache.hadoop.hbase.client.AsyncProcess - #21692403, table=bd_push_device, attempt=29/35 failed=2ops, last exception: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException): Call queue is full on /0.0.0.0:60020, too many items queued ? on lvzhuweng,60020,1581302282313, tracking started null, retrying after=20007ms, replay=2ops
2020-02-15 09:21:59,892 INFO org.apache.hadoop.hbase.client.AsyncProcess - , tableName=bd_push_device
2020-02-15 09:22:19,900 INFO org.apache.hadoop.hbase.client.AsyncProcess - #21692403, table=bd_push_device, attempt=30/35 failed=2ops, last exception: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException): Call queue is full on /0.0.0.0:60020, too many items queued ? on lvzhuweng,60020,1581302282313, tracking started null, retrying after=20106ms, replay=2ops
2020-02-15 09:22:19,902 INFO org.apache.hadoop.hbase.client.AsyncProcess - , tableName=bd_push_device
2020-02-15 09:22:40,008 INFO org.apache.hadoop.hbase.client.AsyncProcess - #21692403, table=bd_push_device, attempt=31/35 failed=2ops, last exception: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.CallQueueTooBigException): Call queue is full on /0.0.0.0:60020, too many items queued ? on lvzhuweng,60020,1581302282313, tracking started null, retrying after=20144ms, replay=2ops
原因:
根据异常日志推测是消费队列(默认长度为10倍的Handler数)打满导致，查看源码ServerRpcConnection类处理请求processRequest()方法：
/**
- @param buf
- Has the request header and the request param and optionally
- encoded data buffer all in this one array.
- @throws IOException
- @throws InterruptedException
*/
protected void processRequest(ByteBuff buf) throws IOException,
InterruptedException {
...if (!this.rpcServer.scheduler.dispatch(new CallRunner(this.rpcServer, call))) {this.rpcServer.callQueueSizeInBytes.add(-1 * call.getSize());this.rpcServer.metrics.exception(RpcServer.CALL_QUEUE_TOO_BIG_EXCEPTION);call.setResponse(null, null, RpcServer.CALL_QUEUE_TOO_BIG_EXCEPTION,"Call queue is full on " + this.rpcServer.server.getServerName() +", too many items queued ?");call.sendResponseIfReady();
}}
得知Server端通过executeRpcCall()方法执行RPC远程调用，CallRunner消费次数超过"hbase.ipc.server.max.callqueue.length"配置值就会引起"too many items queued"异常：
protected boolean executeRpcCall(final ThreadPoolExecutor executor, final AtomicInteger queueSize,
final CallRunner task) {
// Executors provide no offer, so make our own.
int queued = queueSize.getAndIncrement();
if (maxQueueLength > 0 && queued >= maxQueueLength) {queueSize.decrementAndGet();return false;
}executor.execute(new FifoCallRunner(task){@Overridepublic void run() {task.setStatus(RpcServer.getStatus());task.run();queueSize.decrementAndGet();}
});return true;}
public FifoRpcScheduler(Configuration conf, int handlerCount) {
this.handlerCount = handlerCount;
this.maxQueueLength = conf.getInt(RpcScheduler.IPC_SERVER_MAX_CALLQUEUE_LENGTH,handlerCount * RpcServer.DEFAULT_MAX_CALLQUEUE_LENGTH_PER_HANDLER);}
public static final String IPC_SERVER_MAX_CALLQUEUE_LENGTH =
"hbase.ipc.server.max.callqueue.length";
解决方法:
队列 满了, 可以设置大一些。
本文来自互联网用户投稿，文章观点仅代表作者本人，不代表本站立场，不承担相关法律责任。如若转载，请注明出处。 如若内容造成侵权/违法违规/事实不符，请点击【内容举报】进行投诉反馈！