Question

我正在尝试将一个HDFS数据复制到另一个HDFS位置。

我可以使用“distcp”命令实现相同的功能

hadoop distcp hdfs://mySrcip:8020/copyDev/* hdfs://myDestip:8020/copyTest

但我想尝试使用Java Api。经过长时间的搜索找到一个代码并执行。但它没有将我的src文件复制到目的地。

public class TouchFile {

/**
 * @param args
 * @throws Exception 
 */
public static void main(String[] args) throws Exception {
    // TODO Auto-generated method stub
    //create configuration object
    Configuration config = new Configuration();
    config.set("fs.defaultFS", "hdfs://mySrcip:8020/");
    config.set("hadoop.job.ugi", "hdfs");
    /*
     * Distcp
     */
    String sourceNameNode = "hdfs://mySrcip:8020/copyDev";
    String destNameNode = "hdfs://myDestip:8020/copyTest";
    String fileList = "myfile.txt";
    distFileCopy(config,sourceNameNode,destNameNode,fileList);
}
/**
 * Copies files from one cloud to another using Hadoop's distributed copy features. Uses
 * input to build DISTCP configuration settings. 
 *
 * param config Hadoop configuration
 * param sourceNameNode full HDFS path to parent source directory
 * param destNameNode full HDFS path to parent destination directory
 * param fileList Comma separated string of file names in sourceNameNode to be copied to destNameNode
 * returns Elapsed time in milliseconds to copy files
 */
public static long distFileCopy( Configuration config, String sourceNameNode, String destNameNode, String fileList ) throws Exception {
        System.out.println("In dist copy");

    StringTokenizer tokenizer = new StringTokenizer(fileList,",");
    ArrayList<String> list = new ArrayList<>();

    while ( tokenizer.hasMoreTokens() ){
        String file = sourceNameNode + "/" + tokenizer.nextToken();
        list.add( file );
    }

    String[] args = new String[list.size() + 1];
    int count = 0;
    for ( String filename : list ){
        args[count++] = filename;
    }

    args[count] = destNameNode;

    System.out.println("args------>"+Arrays.toString(args));
    long st = System.currentTimeMillis();        
    DistCp distCp=new DistCp(config,null);
    distCp.run(args);   
    return System.currentTimeMillis() - st;

}

}

我做错了什么。请建议

Answer 1

是的，它已经解决了。

这是许可问题。

目标群集应授予用户权限。

无法使用distcp将一个HDFS数据复制到另一个HDFS位置

1 个答案: