Service Monitor未在HBase服务中找到活动的Master

时间:2016-12-15 09:45:05

标签: hadoop hbase cloudera cloudera-cdh cloudera-manager

我使用cloudera CDH 5.8.0创建了一个具有一个主节点和三个从属的集群。经过一些配置工作后,我得到了所有服务的健康,但其中一个:HBase。重新启动后几分钟,健康状况不佳。

Cloudera Manager显示的错误是:“错误:主要摘要:此运行状况测试错误,因为服务监视器未找到活动主服务器”。我检查了服务监视器日志,发现了这个警告:

(7 skipped) Exception in doWork for task: hbase_HBASE_SERVICE_STATE_TASK
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
Thu Dec 15 09:38:30 CET 2016, RpcRetryingCaller{globalStartTime=1481791110299, pause=100, retries=1},     org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2303)
at org.apache.hadoop.hbase.master.MasterRpcServices.getClusterStatus(MasterRpcServices.java:782)
at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55652)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)


at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147)
at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3678)
at org.apache.hadoop.hbase.client.HBaseAdmin.getClusterStatus(HBaseAdmin.java:2382)
at com.cloudera.cmf.cdh5client.hbase.HConnectionImpl.getClusterStatus(HConnectionImpl.java:50)
at com.cloudera.cmon.firehose.polling.hbase.HbaseServiceState.update(HbaseServiceState.java:158)
at com.cloudera.cmon.firehose.polling.hbase.HbaseServiceStateFetcher.doWork(HbaseServiceStateFetcher.java:42)
at com.cloudera.cmon.firehose.polling.AbstractHConnectionClientTask.doWorkWithClientConfig(AbstractHConnectionClientTask.java:95)
at com.cloudera.cmon.firehose.polling.AbstractHConnectionClientTask.doWorkWithClientConfig(AbstractHConnectionClientTask.java:26)
at com.cloudera.cmon.firehose.polling.AbstractCdhWorkUsingClientConfigs.doWork(AbstractCdhWorkUsingClientConfigs.java:45)
at com.cloudera.cmon.firehose.polling.CdhTask$InstrumentedWork.doWork(CdhTask.java:230)
at com.cloudera.cmf.cdhclient.util.ImpersonatingTaskWrapper.runTask(ImpersonatingTaskWrapper.java:72)
at com.cloudera.cmf.cdhclient.util.ImpersonatingTaskWrapper.access$000(ImpersonatingTaskWrapper.java:21)
at com.cloudera.cmf.cdhclient.util.ImpersonatingTaskWrapper$1.run(ImpersonatingTaskWrapper.java:107)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at com.cloudera.cmf.cdh5client.security.UserGroupInformationImpl.doAs(UserGroupInformationImpl.java:41)
at com.cloudera.cmf.cdhclient.util.ImpersonatingTaskWrapper.doWork(ImpersonatingTaskWrapper.java:103)
at com.cloudera.cmf.cdhclient.CdhExecutor$1.call(CdhExecutor.java:125)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2303)
at org.apache.hadoop.hbase.master.MasterRpcServices.getClusterStatus(MasterRpcServices.java:782)
at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55652)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)

at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1219)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:300)
at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.getClusterStatus(MasterProtos.java:46458)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$5.getClusterStatus(ConnectionManager.java:2027)
at org.apache.hadoop.hbase.client.HBaseAdmin$28.call(HBaseAdmin.java:2386)
at org.apache.hadoop.hbase.client.HBaseAdmin$28.call(HBaseAdmin.java:2382)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
... 22 more

有没有一种解决这个问题的已知方法?我将主节点设置为HBase Master,将从设备设置为HBase RegionServers。

2 个答案:

答案 0 :(得分:2)

问题在于Cloudera Management Monitor Service,而不在于Hbase本身。我所做的是重新启动Cloudera Management Monitor Service,然后重新启动hbase。之后一切似乎都很好。

答案 1 :(得分:0)

检查hbase master的日志文件。 以我为例,hbase没有足够的权限来操作HDFS上的“ / tmp”目录,因此我对其进行了更改,问题就消失了。