Question

我们有两个集群，其中hbase为0.94，hadoop为1.04，hbase为0.98，hadoop为2.4 我已经在0.94快照的表格中创建了快照，并希望将其迁移到hbase 0.98的群集。

在0.98群集上运行此命令后：

hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapshot-name -copy-from webhdfs://hadoops-master:9000/hbase -copy-to hdfs://solr1:8020/hbase

我明白了：

    Exception in thread "main" org.apache.hadoop.hbase.snapshot.ExportSnapshotException: Failed to copy the snapshot directory: from=webhdfs://hadoops-master:9000/hbase/.hbase-snapshot/snapshot-name to=hdfs://solr1:8020/hbase/.hbase-snapshot/.tmp/snapshot-name
    at org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:916)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:1000)
    at org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:1004)
Caused by: java.net.SocketException: Unexpected end of file from server
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
    at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getResponse(WebHdfsFileSystem.java:596)
    at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.run(WebHdfsFileSystem.java:530)
    at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:417)
    at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:630)
    at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:641)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
    at org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:914)
    ... 3 more

Answer 1

Lars Hofhansl本人（HBase的主要提交人和0.94的维护者）表示不支持0.94和0.98的出口。所以你现在可能会继续这样做。

以下是今天下午他的帖子的摘录：

It's tricky from multiple angles:- replication between 0.94 and 0.98 does not work (there's a gateway process that supposedly does that, but it's not known to be very reliable)- snapshots cannot be exported from 0.94 and 0.98

来源：hbase用户今天12/15/14的邮件列表

更新在HBase邮件列表上有一个报告，一个用户能够找到导出的方法。一条信息是：

If exporting from an HBase 0.94 cluster to an HBase 0.98 cluster, you will need to use the webhdfs protocol (or possibly hftp, though I couldn’t get that to work). You also need to manually move some files around because snapshot layouts have changed. Based on the example above, on the 0.98 cluster do the following:

   check whether any imports already exist for the table:

  hadoop fs -ls /apps/hbase/data/archive/data/default

我在这里重现整个讨论是不正确的：包含整个血腥细节的nabble链接是：

http://apache-hbase.679495.n3.nabble.com/0-94-going-forward-td4066883.html

Answer 2

我正在深入研究这个问题，它更多地与底层HDFS有关。

一旦编写了流（在我的情况下为distcp），就会调用close：

public void close() throws IOException {
try {
      super.close();
    } finally {
      try {
        validateResponse(op, conn, true);
      } finally {
        conn.disconnect();
      }
    }
  }

其中在验证响应调用中失败（可能是连接的另一端已关闭）。

这可能是由于HDFS 1.0和2.4之间的不兼容性所致。

Answer 3

我用jarjar https://code.google.com/p/jarjar/重新打包了hadoop和hbase libs 它需要修复一些类名问题。然后我写了一个mapreduce copyTable作业。它从94 culster读取行并写入98簇。这是代码： https://github.com/fiserro/copy-table-94to98

感谢github.com/falsecz的想法和帮助！

hbase ExportSnapshot从0.94到0.98

3 个答案: