Question

我正在使用oozie通过shell脚本运行sparkr。当我开始工作时，我面临许可问题：

Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=yarn, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
ERROR Utils: Uncaught exception in thread delete Spark local dirs
java.lang.NullPointerException
Exception in thread "delete Spark local dirs" java.lang.NullPointerException

整体日志..

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p0.2/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/mntc/yarn/nm/filecache/2452/sparkr-  assembly-0.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/09/08 01:39:11 INFO SparkContext: Running Spark version 1.3.0
15/09/08 01:39:13 INFO SecurityManager: Changing view acls to: yarn
15/09/08 01:39:13 INFO SecurityManager: Changing modify acls to: yarn
15/09/08 01:39:13 INFO SecurityManager: SecurityManager: authentication 
disabled; ui acls disabled; users with view permissions: Set(yarn); users with modify permissions: Set(yarn)
15/09/08 01:39:13 INFO Slf4jLogger: Slf4jLogger started
15/09/08 01:39:13 INFO Remoting: Starting remoting
15/09/08 01:39:14 INFO Remoting: Remoting started; listening on addresses :  
  /mnt/yarn/nm/usercache/karun/appcache
  /application_1437539731669_0786/blockmgr-1760ec19-b1de-4bcc-9100- b2c1364b54c8 
  /mntc/yarn/nm/usercache/karun/appcache/application_1437539731669_0786/    
   blockmgr-f57c89eb-4a4b-4fd5-9796-ca3c3a7f2c6f
 15/09/08 01:39:14 INFO DiskBlockManager: Created local directory at     
  15/09/08 01:39:14 INFO MemoryStore: MemoryStore started with capacity 265.4 MB
  15/09/08 01:39:15 INFO Utils: Successfully started service 'SparkUI' on port 4040.
    15/09/08 01:39:16 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (6610 MB per container)
    15/09/08 01:39:16 INFO Client: Preparing resources for our AM container
    createSparkContext on edu.berkeley.cs.amplab.sparkr.RRDD failed with  java.lang.reflect.InvocationTargetException
  Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=yarn, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
  Exception in thread "delete Spark local dirs" java.lang.NullPointerException
  Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

我不知道如何解决这个问题。非常感谢。

Answer 1

问题可能是因为您在HDFS中没有此类用户“yarn”。有两种可能的解决方案。

在HDFS上创建此类用户，并授予他访问所需资源的权限。
更简单，只需在hdfs用户下执行您的工作（或者在hdfs上执行任何操作）。在oozie属性文件中设置user.name = hdfs。 ozie doc

为什么执行SparkR作业会使用Oozie拒绝权限？

1 个答案: