UDF(Java):HDFS

时间:2017-05-05 08:06:19

标签: java hive hdfs udf

我写一个hive-UDTF来通过在HDFS上加载.dat文件来解析ip地址,但遇到错误:

java.io.FileNotFoundException: hdfs:/THE_IP_ADDRESS:9000/tmp/ip_20170204.dat (Permission denied) 

但实际上,dfs目录/ tmp和.data文件都具有完全访问权限:777,我无法修改配置以禁用dfs权限。

我的UDTF中读取文件的行:

IP.load("hdfs://THE_IP_ADDRESS:9000/tmp/ip_20170204.dat");

和静态方法.load()

public static void load(String filename) {
        ipFile = new File(filename);
        load();
        if (enableFileWatch) {
            watch();
        }
    }


private static void load() {
    lastModifyTime = ipFile.lastModified();
    FileInputStream fin = null;
    lock.lock();
    try {
        dataBuffer = ByteBuffer.allocate(Long.valueOf(ipFile.length()).intValue());
        fin = new FileInputStream(ipFile);
        int readBytesLength;
        byte[] chunk = new byte[4096];
        while (fin.available() > 0) {
            readBytesLength = fin.read(chunk);
            dataBuffer.put(chunk, 0, readBytesLength);
        }
        dataBuffer.position(0);
        int indexLength = dataBuffer.getInt();
        byte[] indexBytes = new byte[indexLength];
        dataBuffer.get(indexBytes, 0, indexLength - 4);
        indexBuffer = ByteBuffer.wrap(indexBytes);
        indexBuffer.order(ByteOrder.LITTLE_ENDIAN);
        offset = indexLength;

        int loop = 0;
        while (loop++ < 256) {
            index[loop - 1] = indexBuffer.getInt();
        }
        indexBuffer.order(ByteOrder.BIG_ENDIAN);
    } catch (IOException ioe) {
        ioe.printStackTrace();
    } finally {
        try {
            if (fin != null) {
                fin.close();
            }
        } catch (IOException e){
            e.printStackTrace();
        }
        lock.unlock();
    }
}

0 个答案:

没有答案