Question

我对Hadoop很新。但是，我已经能够在我的服务器上以群集模式成功设置带有Java 7的hadoop 2.7.3。一切都很好。

但是，当我尝试切换到Java 8并启动dfs时，出现错误：

Exception in thread "main" java.io.IOException: failure to login
    at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:824)
    at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761)
    at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634)
    at org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:315)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at org.apache.hadoop.hdfs.tools.GetConf.main(GetConf.java:332)
Caused by: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
    at com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:71)
    at com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:133)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
    at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
    at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
    at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:799)
    at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761)
    at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634)
    at org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:315)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at org.apache.hadoop.hdfs.tools.GetConf.main(GetConf.java:332)

    at javax.security.auth.login.LoginContext.invoke(LoginContext.java:856)
    at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
    at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
    at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:799)
    ... 6 more

似乎hadoop碰巧无法使用我的工具用户名登录。我尝试从其他来源查找并遵循他们的指导，例如添加环境变量HADOOP_USER_NAME，删除和重置整个datanode和namenode等等。但是，在这种情况下，它们都不适用于我。

关于这个问题的奇怪之处在于，当我尝试切换回Java 7. Hadoop像以前一样对我很好。但是，我不认为应该存在问题，因为Java版本之间存在差异，因为Hadoop基于其文档与它们兼容。此外，已在群集中设置了受信任的ssh。

下面是我的.bashrc配置fyi：

export JAVA_HOME="/scratch/dsat_server/jdk1.8.0_121"
export HADOOP_PREFIX="/scratch/dsat_server/hadoop-2.7.3"
export HADOOP_COMMON_LIB_NATIVE_DIR="$HADOOP_PREFIX/lib/native"
export HADOOP_CONF_DIR="$HADOOP_PREFIX/conf"
export HADOOP_OPTS="-Djava.net.preferIPv4Stack=truei-Djava.library.path=$HADOOP_PREFIX/lib"

Answer 1

我从使用Java 8的docker容器运行hbase客户端时遇到了同样的问题。它显然是由类com.sun.security.auth.module.UnixLoginModule引起的，它使用本机调用来获取unix用户名。在我的例子中，它没有映射到docker中，并且该类抛出NullPointerException。这不是hadoop本身的错误。

为了指示hadoop绕过OS用户名的查找，我可以在所有初始化之前添加以下代码行：

UserGroupInformation.setLoginUser(UserGroupInformation.createRemoteUser("hduser"));

在您的情况下，您正在运行服务器，因此您注入代码的选项有限。相反，有两种选择：

尝试使用IBM JDK
尝试调试worker（$ whoami）上的OS用户设置。如果它显示“无法找到用户ID XXXX的名称”，请检查/ etc / passwd设置

Answer 2

从IDE运行spark代码时遇到相同的错误。

Exception in thread "main" java.io.IOException: failure to login
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:822)
        at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:774)
        at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
        at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2464)
        at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2464)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2464)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:292)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
        at SimpleApp$.main(SimpleApp.scala:17)
        at SimpleApp.main(SimpleApp.scala)
Caused by: javax.security.auth.login.LoginException:
java.lang.NullPointerException: invalid null input: name

决议在主要代码中添加了以下声明。

UserGroupInformation.setLoginUser(UserGroupInformation.createRemoteUser("vyxx"))

Answer 3

我发现，如果我确保为jenkins用户正确配置了docker映像，则无需在代码中采取任何特殊措施。我用于为基于Debian / Ubuntu的图像的jenkins用户设置图像的代码是：

# Add Jenkins user
groupadd --gid 1000 jenkins
useradd --uid 1000 --gid jenkins --shell /bin/bash --home-dir /var/jenkins_home jenkins
mkdir /var/jenkins_home
chown 1000:1000 /var/jenkins_home
echo 'jenkins ALL=NOPASSWD: ALL' >> /etc/sudoers.d/50-jenkins
echo 'Defaults    env_keep += "DEBIAN_FRONTEND"' >> /etc/sudoers.d/env_keep

Hadoop IOException登录失败

3 个答案: