使用kerberos安全的Sqoop无法在cron中工作,抛出GSSException

时间:2017-11-24 03:22:29

标签: hadoop hive kerberos sqoop

当我从边缘节点运行sqoop作业时,它工作正常并且能够从oracle中提取数据,但是当我在crontab中安排相同的作业时,它会抛出kerberos安全错误。

我在hortonworks网站上发现了相同的问题(请参阅链接),但没有任何有效答案。

https://community.hortonworks.com/questions/61856/kerberos-ticket-error-in-a-cron-job.html

我在cron作业开始之前有有效的kerberos票,有效期为10小时,但是当我在5分钟内安排cron作业时,它会抛出kerberos错误。

请建议在cron job中启动sqoop导入之前需要遵循的步骤。

以下是详细的错误消息:

  

17/11/23 11:24:17错误工具.ImportTool:遇到IOException正在运行导入作业:java.io.IOException:本地异常失败:java.io.IOException:javax.security.sasl.SaslException:< / p>      

GSS启动失败[由GSS异常引起:未提供有效凭据(机制级别:无法找到任何Kerberos tgt)];

     

主机详细信息:本地主机为:&#34; ps2pr028380.express-scripts.com/10.221.171.22" ;;目的地主机是:&#34; ps2pr028377.express-scripts.com&#34;:8020;         在org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:782)         在org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1558)         在org.apache.hadoop.ipc.Client.call(Client.java:1498)         在org.apache.hadoop.ipc.Client.call(Client.java:1398)         在org.apache.hadoop.ipc.ProtobufRpcEngine $ Invoker.invoke(ProtobufRpcEngine.java:233)         at com.sun.proxy。$ Proxy11.getDelegationToken(Unknown Source)         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken(ClientNamenodeProtocolTranslatorPB.java:980)         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:498)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:291)         在org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:203)         在org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:185)         在com.sun.proxy。$ Proxy12.getDelegationToken(未知来源)         在org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:1041)         at org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:1688)         在org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:549)         在org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:527)         在org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2400)         at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140)         at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)         at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)         at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:142)         在org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:266)         在org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)         在org.apache.hadoop.mapreduce.Job $ 10.run(Job.java:1290)         在org.apache.hadoop.mapreduce.Job $ 10.run(Job.java:1287)         at java.security.AccessController.doPrivileged(Native Method)         在javax.security.auth.Subject.doAs(Subject.java:422)         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)         在org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)         在org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)         在org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:200)         在org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:173)         在org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:270)         在org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:748)         at org.apache.sqoop.manager.OracleManager.importQuery(OracleManager.java:454)         在org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:509)         在org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615)         在org.apache.sqoop.Sqoop.run(Sqoop.java:147)         在org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)         在org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)         在org.apache.sqoop.Sqoop.runTool(Sqoop.java:225)         在org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)         在org.apache.sqoop.Sqoop.main(Sqoop.java:243)   引起:java.io.IOException:javax.security.sasl.SaslException:GSS启动失败[由GSSException引起:未提供有效凭据(机制级别:无法找到任何Kerberos tgt)]         在org.apache.hadoop.ipc.Client $ Connection $ 1.run(Client.java:720)         at java.security.AccessController.doPrivileged(Native Method)         在javax.security.auth.Subject.doAs(Subject.java:422)         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)         at org.apache.hadoop.ipc.Client $ Connection.handleSaslConnectionFailure(Client.java:683)         在org.apache.hadoop.ipc.Client $ Connection.setupIOstreams(Client.java:770)         在org.apache.hadoop.ipc.Client $ Connection.access $ 3200(Client.java:397)         在org.apache.hadoop.ipc.Client.getConnection(Client.java:1620)         在org.apache.hadoop.ipc.Client.call(Client.java:1451)         ......还有43个   引起:javax.security.sasl.SaslException:GSS启动失败[由GSSException引起:未提供有效凭据(机制级别:无法找到任何Kerberos tgt)]         在com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)         在org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)         在org.apache.hadoop.ipc.Client $ Connection.setupSaslConnection(Client.java:595)         在org.apache.hadoop.ipc.Client $ Connection.access $ 2000(Client.java:397)         在org.apache.hadoop.ipc.Client $ Connection $ 2.run(Client.java:762)         在org.apache.hadoop.ipc.Client $ Connection $ 2.run(Client.java:758)         at java.security.AccessController.doPrivileged(Native Method)         在javax.security.auth.Subject.doAs(Subject.java:422)         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)         在org.apache.hadoop.ipc.Client $ Connection.setupIOstreams(Client.java:757)         ......还有46个   引起:GSSException:未提供有效凭据(机制级别:无法找到任何Kerberos tgt)         at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)         at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)         at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)         at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)         at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)         at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)         在com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)         ......还有55个

1 个答案:

答案 0 :(得分:1)

我解决了我自己的问题:我在我的脚本中添加了以下注释,并在sqoop启动之前调用了它。

 kinit -kt /home/userName/userName.keytab userName@PROD.DATAHUB.LOCAL

我们正在使用hortonwork集群,因此他们有另一个解决此问题的方法,请找到下面的其他解决方案链接

https://community.hortonworks.com/questions/149168/sqoop-with-kerberos-security-not-working-in-cron-t.html