错误：hbase：meta表不一致

时间：2018-10-06 04:56:35

标签： hadoop hbase

在HBASE中创建表时出错。 ”错误：java.io.IOException：表命名空间管理器尚未准备好，请尝试     再过一遍。”

hcbk -fix显示 错误：在任何区域都找不到hbase：meta。

hbase shell会话全新启动后出现错误。启动期间，主日志中未报告任何错误。 Hbase的上一个会话正常关闭，但没有关闭zookeeper（怀疑这是造成元表损坏的原因）。

我能够列出之前创建的表

hbase(main):001:0> list
TABLE
IDX_STOCK_SYMBOL
Patient
STOCK_SYMBOL
STOCK_SYMBOL_BKP
SYSTEM.CATALOG
SYSTEM.FUNCTION
SYSTEM.SEQUENCE
SYSTEM.STATS
8 row(s) in 1.7930 seconds

创建一个名为custmaster的表

    hbase(main):002:0> create 'custmaster', 'customer'

    ERROR: java.io.IOException: Table Namespace Manager not ready yet, try 
    again later
    at org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:3179)
    at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1735)
    at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1774)
    at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40470)
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027)
    at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
    at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

解决方法：运行hbck来识别不一致之处

    [hduser@master ~]$ hbase hbck
    >Version: 0.98.4-hadoop2
    >Number of live region servers: 2
    >Number of dead region servers: 0
    >Master: master,60000,1538793456542
    >Number of backup masters: 0
    >Average load: 0.0
    >Number of requests: 11
    >Number of regions: 0
    >Number of regions in transition: 1
    >
    >ERROR: META region or some of its attributes are null.
    >ERROR: hbase:meta is not found on any region.
    >ERROR: hbase:meta table is not consistent. Run HBCK with proper fix options to fix hbase:meta inconsistency. Exiting...
    .
    .
    .
    >Summary:                >
    >3 inconsistencies detected.
    >Status: INCONSISTENT

Ran hbck wih-details选项可识别涉及的表

    [hduser@master ~]$ hbase hbck -details
    >ERROR: META region or some of its attributes are null.
    >ERROR: hbase:meta is not found on any region.
    >ERROR: hbase:meta table is not consistent. Run HBCK with proper fix options to fix hbase:meta inconsistency. Exiting...
    >Summary:
    >3 inconsistencies detected.
    >Status: INCONSISTENT

-details的输出清楚地表明在任何区域都找不到该元。

尝试运行命令hbase hbck -fixMeta，但返回的结果与上述相同 因此尝试了 hbase hbck -fix

此命令运行了一段时间，提示是“正在尝试解决hbase：meta ..的问题。” ，并导致以下错误

    [hduser@master ~]$ hbase hbck -fix

    Version: 0.98.4-hadoop2
    Number of live region servers: 2
    Number of dead region servers: 0
    Master: master,60000,1538793456542
    Number of backup masters: 0
    Average load: 0.0
    Number of requests: 19
    Number of regions: 0
    Number of regions in transition: 1
    ERROR: META region or some of its attributes are null.
    ERROR: hbase:meta is not found on any region.
    Trying to fix a problem with hbase:meta..
    2018-10-06 09:01:03,424 INFO  [main] client.HConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
    2018-10-06 09:01:03,425 INFO  [main] client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x166473bbe720005
    2018-10-06 09:01:03,432 INFO  [main] zookeeper.ZooKeeper: Session: 0x166473bbe720005 closed
    2018-10-06 09:01:03,432 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
    Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions:
    Sat Oct 06 08:52:13 IST 2018, org.apache.hadoop.hbase.client.RpcRetryingCaller@18920cc, org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
            at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2416)
            at org.apache.hadoop.hbase.master.HMaster.assignRegion(HMaster.java:2472)
            at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40456)
            at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027)
            at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
            at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)

    Sat Oct 06 08:52:13 IST 2018, org.apache.hadoop.hbase.client.RpcRetryingCaller@18920cc, org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.PleaseHoldException): org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
            at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2416)
            at org.apache.hadoop.hbase.master.HMaster.assignRegion(HMaster.java:2472)
            at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40456)
            at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027)
            at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
            at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)

请帮助我解决该问题？ 在此先感谢!!

1 个答案:

答案 0 :(得分：0)

我还没有检查NameNode和Datanode日志。但是当我检查出真正的问题时发现HDFS中的文件损坏。

运行 hadoop fsck / 检查文件系统的运行状况。 

    [hduser@master ~]$ hadoop fsck /
    DEPRECATED: Use of this script to execute hdfs command is deprecated.
    Instead use the hdfs command for it.

    18/10/06 09:52:00 WARN util.NativeCodeLoader: Unable to load native-hadoop libr
    ary for your platform... using builtin-java classes where applicable
    Connecting to namenode via http://master:50070/fsck?ugi=hduser&path=%2F
    FSCK started by hduser (auth:SIMPLE) from /192.168.1.11 for path / at Sat Oct 0
    6 09:52:02 IST 2018
    ...............................................................................
    ..
    /user/hduser/hbase/.hbck/hbase-1538798774320/data/hbase/meta/1588230740/info/35
    9783d4cd07419598264506bac92dcf: CORRUPT blockpool BP-1664228054-192.168.1.11-15
    35828595216 block blk_1073744002

    /user/hduser/hbase/.hbck/hbase-1538798774320/data/hbase/meta/1588230740/info/35                                                   9783d4cd07419598264506bac92dcf: MISSING 1 blocks of total size 3934 B.........
    /user/hduser/hbase/data/default/IDX_STOCK_SYMBOL/a27db76f84487a05f3e1b8b74c13fa
    78/0/c595bf49443f4daf952df6cdaad79181: CORRUPT blockpool BP-1664228054-192.168.
    1.11-1535828595216 block blk_1073744000

    /user/hduser/hbase/data/default/IDX_STOCK_SYMBOL/a27db76f84487a05f3e1b8b74c13fa
    78/0/c595bf49443f4daf952df6cdaad79181: MISSING 1 blocks of total size 1354 B...
    .........
    ...
    /user/hduser/hbase/data/default/SYSTEM.CATALOG/d63574fdd00e8bf3882fcb6bd53c3d83
    /0/dcb68bbb5e394d19b06db7f298810de0: CORRUPT blockpool BP-1664228054-192.168.1.
    11-1535828595216 block blk_1073744001

    /user/hduser/hbase/data/default/SYSTEM.CATALOG/d63574fdd00e8bf3882fcb6bd53c3d83
    /0/dcb68bbb5e394d19b06db7f298810de0: MISSING 1 blocks of total size 2283 B.....                                                   ......................Status: CORRUPT
     Total size:    4232998 B
     Total dirs:    109
     Total files:   129
     Total symlinks:                0
     Total blocks (validated):      125 (avg. block size 33863 B)
      ********************************
      UNDER MIN REPL'D BLOCKS:      3 (2.4 %)
      dfs.namenode.replication.min: 1
      CORRUPT FILES:        3
      MISSING BLOCKS:       3
      MISSING SIZE:         7571 B
      CORRUPT BLOCKS:       3
      ********************************
     Minimally replicated blocks:   122 (97.6 %)
     Over-replicated blocks:        0 (0.0 %)
     Under-replicated blocks:       0 (0.0 %)
     Mis-replicated blocks:         0 (0.0 %)
     Default replication factor:    2
     Average block replication:     1.952
     Corrupt blocks:                3
     Missing replicas:              0 (0.0 %)
     Number of data-nodes:          2
     Number of racks:               1
    FSCK ended at Sat Oct 06 09:52:02 IST 2018 in 66 milliseconds


    The filesystem under path '/' is CORRUPT

然后运行 hdfs hbck -delete 选项以删除损坏的文件并解决此问题。

有关清理hdfs f / s的详细说明，请参见-> How to fix corrupt HDFS FIles

相关问题
最新问题