Question

我有一个Java程序，它将文件从Unix复制到hdfs。运行正常，但是我正在寻找一个模拟帐户来运行并复制文件。

输入：除了输入文件和目标hdfs目录路径之外，另一个输入应该是包含帐户，keytab目录，域的属性文件

请让我知道前进的最佳方法。

我目前正在探索使用shell首先发出kinit命令然后运行jar

我还从-https://henning.kropponline.de/2016/02/14/a-secure-hdfs-client-example/

了解有关Jaas以及如何在Java本身中完成此操作的信息。

需要输入和可用选项的任何参考。

我复制文件的Java程序如下：

public class FileCopy implements Runnable {

@Option(names = {"-i","--input"}, required=true, description="file name to copy to hadoop")
String input;

@Option(names = {"-o","--output"}, required=true, description="hdfs directory path to be copied into")
String output;


public void run() {


    Properties hadoop_properties = new Properties();
    HdfsFileDeploy hdfsFileDeploy = new HdfsFileDeploy();

    try {
        hadoop_properties.load(FileCopy.class.getClassLoader().getResourceAsStream("hadoop.properties"));
    } catch (IOException e) {
        e.printStackTrace();
    }


    FileSystem fs = hdfsFileDeploy.configureFilesystem(hadoop_properties.getProperty("coreSitePath"),hadoop_properties.getProperty("hdfsSitePath"));


    String status = hdfsFileDeploy.writeToHDFS(fs,input,output);

    if (status == "SUCCESS") {
            System.out.println("completed copying");
    } else {
        System.out.println("copying error");
    }

    hdfsFileDeploy.closeFileSystem(fs);

}

public static void main(String[] args) throws IOException {


    CommandLine.run(new FileCopy(), args);

}

}

公共类HdfsFileDeploy {

   public FileSystem configureFilesystem(String coreSitePath, String hdfsSitePath) {

        FileSystem fileSystem = null;

        try {

            Configuration conf = new Configuration();
            Path hdfsCoreSitePath = new Path(coreSitePath);
            Path hdfsHDFSSitePath = new Path(hdfsSitePath);
            conf.addResource(hdfsCoreSitePath);
            conf.addResource(hdfsHDFSSitePath);


            fileSystem = FileSystem.get(conf);
            System.out.println(fileSystem);
            return fileSystem;

        } catch (Exception ex) {

            ex.printStackTrace();
            return fileSystem;
        }
    }


   public void closeFileSystem(FileSystem fileSystem) {

        try {
            fileSystem.close();
        } catch (Exception ex) {
                System.out.println("Unable to close Hadoop filesystem : " + ex);

        }
   }

   // 

    public String writeToHDFS(FileSystem fileSystem, String sourcePath, String destinationPath) {

        String failure = "FAILURE";
        String success = "SUCCESS";
        Boolean doNotDelSrc = false;
        Boolean overwrite = true;

        try {
            Path inputPath = new Path(sourcePath);
            Path outputPath = new Path(destinationPath);

            if(!fileSystem.exists(outputPath)) {
                System.out.println("Output path " + outputPath + " does not exist. Creating outputPath directory now..");
                if (fileSystem.mkdirs(outputPath)) {
                    System.out.println("Output path " + outputPath + " created...");
                }
            }

            System.out.println("about to copy from " + inputPath + " to " + outputPath);
            fileSystem.copyFromLocalFile(doNotDelSrc, overwrite, inputPath, outputPath);
            return success;

        } catch (IOException ex) {

            System.out.println("Some exception occurred while writing file to hdfs");
            ex.printStackTrace();
            return failure;
        }
    }

}

Input1：输入文件 Input2：目标hdfs目录参考输入：包含帐户，域，密钥表路径的文件（例如yaml）。

jar应该模拟输入文件并将其复制到目标hdfs目录。

如何最好地模拟hadoop上的用户帐户

0 个答案: