Question

由于对hdfs:///user/hdfs文件夹的访问限制，在hdp 3.1.4上运行h2o（http://h2o-release.s3.amazonaws.com/h2o/rel-yau/5/h2o-3.26.0.5-hdp3.1.zip）在启动时出现错误

[root@HW005 h2o-3.26.0.5-hdp3.1]# hadoop jar h2odriver.jar -nodes 4 -mapperXmx 6g
Determining driver host interface for mapper->driver callback...
    [Possible callback IP address: 172.18.4.83]
    [Possible callback IP address: 127.0.0.1]
Using mapper->driver callback IP address and port: 172.18.4.83:37342
(You can override these with -driverif and -driverport/-driverportrange and/or specify external IP using -extdriverif.)
Memory Settings:
    mapreduce.map.java.opts:     -Xms6g -Xmx6g -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Dlog4j.defaultInitOverride=true
    Extra memory percent:        10
    mapreduce.map.memory.mb:     6758
Hive driver not present, not generating token.
19/09/17 10:38:17 INFO client.RMProxy: Connecting to ResourceManager at hw001.co.local/172.18.4.46:8050
19/09/17 10:38:17 INFO client.AHSProxy: Connecting to Application History server at hw002.co.local/172.18.4.47:10200
ERROR: Permission denied: user=root, access=WRITE, inode="/user":hdfs:hdfs:drwxr-xr-x

这似乎是一个要求，这很奇怪，因为我想根据用例以不同的用户身份运行h2o，而我不认为仅允许访问hdfs用户（HDP默认HDFS）是不正确的管理员用户）HDFS主文件夹中才能执行此操作。谁能解释这里发生的事情以及通常如何处理？

Answer 1

如何在非内核化的Hadoop集群中管理模拟，用于...
*为任意Hadoop用户创建HDFS HomeDir
*在该用户下运行作业（它将使用HomeDir处理临时文件）

## create HDFS HomeDir for new user, with "hdfs" privileged account
## note the workaround for the bug in "chmod" parser which
##  fails on "=<nothing>" in most Hadoop versions
export HADOOP_USER_NAME=hdfs
hdfs dfs -mkdir -p               /user/zorro
hdfs dfs -chown zorro:zorro      /user/zorro
hdfs dfs -chmod u=rwx,g=rx,o-rwx /user/zorro
unset HADOOP_USER_NAME

export HADOOP_USER_NAME=zorro
# just to check who's there
hdfs groups

run-my-H2O-job-on-command-line
unset HADOOP_USER_NAME

H2O Hadoop是否需要访问用户hdfs的HDFS主文件夹？

1 个答案: