火花与凤凰一起提交

时间:2017-06-30 13:39:59

标签: apache-spark hbase phoenix

我正在尝试使用安全群集上的spark提交连接到凤凰我有这个例外

  

ConnectionQueryServicesImpl:尝试连接到安全群集   with keytab:/ hbase-secure 17/06/29 11:54:44 ERROR Executor:Exception   在阶段0.0(TID 0)的任务0.0中java.sql.SQLException:错误103   (08004):无法建立连接。

在我的程序中我想保存到phoenix一个数据帧(生成异常的程序部分)我已经编译并从我的intelliji保存到phoenix但是它不能用spark提交,即使有本地主

这是生成异常的代码的一部分

       df.write
        .format("org.apache.phoenix.spark")
        .mode("overwrite")
        .option("table", output)
        .option("zkUrl", "sv005689.info.ratp:2181") 
        .save()

这是我的火花提交命令

spark-submit --class TEST.PHOENIX --master local --executor-memory 10G --num-executors 50 --conf spark.ui.port=14040  spark-assembly-0.0.2.jar

这是我的例外

17/06/29 11:54:43 INFO ConnectionQueryServicesImpl: Trying to connect to a secure cluster with keytab:/hbase-secure
17/06/29 11:54:44 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.sql.SQLException: ERROR 103 (08004): Unable to establish connection.
        at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:422)
        at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection(ConnectionQueryServicesImpl.java:392)
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.access$300(ConnectionQueryServicesImpl.java:211)
        at org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2269)
        at org.apache.phoenix.query.ConnectionQueryServicesImpl$13.call(ConnectionQueryServicesImpl.java:2248)
        at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:78)
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2248)
        at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:233)
        at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:135)
        at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:202)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:208)
        at org.apache.phoenix.mapreduce.util.ConnectionUtil.getConnection(ConnectionUtil.java:98)
        at org.apache.phoenix.mapreduce.util.ConnectionUtil.getOutputConnection(ConnectionUtil.java:82)
        at org.apache.phoenix.mapreduce.util.ConnectionUtil.getOutputConnection(ConnectionUtil.java:70)
        at org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil.getUpsertColumnMetadataList(PhoenixConfigurationUtil.java:230)
        at org.apache.phoenix.spark.DataFrameFunctions$$anonfun$2.apply(DataFrameFunctions.scala:45)
        at org.apache.phoenix.spark.DataFrameFunctions$$anonfun$2.apply(DataFrameFunctions.scala:41)
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$22.apply(RDD.scala:717)
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$22.apply(RDD.scala:717)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
        at org.apache.spark.scheduler.Task.run(Task.scala:89)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Login failure for 2181 from keytab /hbase-secure: javax.security.auth.login.LoginException: Unable to obtain password from user

        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:987)
        at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:280)
        at org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:386)
        at org.apache.hadoop.hbase.security.User.login(User.java:253)
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection(ConnectionQueryServicesImpl.java:380)
        ... 27 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user

        at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897)
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)

有人可以帮我解决这个问题  谢谢你的帮助

0 个答案:

没有答案