线程“主”中的异常java.lang.IllegalStateException:org.apache.kudu.shaded.com.google.common.base.Preconditions.checkState

时间:2019-09-10 19:07:43

标签: scala apache-spark kerberos apache-kudu

我正在尝试在Apache Kudu上运行我的代码。虽然在我的本地kudu设置上运行正常,但我在同一台计算机上拥有主服务器和平板电脑服务器。现在,通过在Cloudera Hadoop env上通过rpm软件包安装手动安装Kudu,我得到了例外。

我正在尝试通过spark加载一个csv文件并坚持到Kudu。我已经在工作状态下设置了以下内容:

在不同的节点上安装了Kudu主机和平板电脑(它们正在运行) 在所有群集节点上都设置了新用户(kudu)的kereberos票证。 (但是,只有一台母版和一台平板电脑,只需对其中两个进行初始化即可)。针对每个组件(例如spark,hive等)都制定了护林员政策,以便向另一个组件/从另一个组件提交作业

/**
    * Storing the dataframe into Kudu
    *
    * @param df the dataframe to store.
    */
  def store(df: DataFrame): Unit = {
    // Init configuration.
    log.info("kuduMaster: " + config.kuduMaster)
    val kuduContext = new KuduContext(config.kuduMaster, config.sparkSession.sparkContext)
    val primaryKeys = SchemaMapper.primaryKeys(config.tableName).get

    // Create Table if non existing
    if (!kuduContext.tableExists(config.tableName)) {
      log.info("Create Table, tableName: " + config.tableName)
      kuduContext.createTable(
        config.tableName, df.schema, primaryKeys, new CreateTableOptions()
          .setNumReplicas(config.kuduNumReplicas)
          .addHashPartitions(primaryKeys.asJava, config.kuduBuckets))
    }

Spark插入Kudu时出现异常。成功启动Kerberos主体后: 以下是每次出现的例外情况。我试图从kudu安装所在的Hadoop的同一节点上并通过kudu用户运行它,但是它不起作用。

19/09/10 18:33:43 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.
19/09/10 18:33:43 INFO CoreUtils$: Elapsed time [KuduLoader load csv->df]: 1.727817664s
19/09/10 18:33:43 INFO KuduLoader$: kuduMaster: kudu://mynode.ic.company.net:7051
19/09/10 18:33:43 INFO KuduContext: Logging in as principal kudu@mycluster.IC.COMPANY.NET with keytab /home/kudu/kudu.keytab
Exception in thread "main" java.lang.IllegalStateException
        at org.apache.kudu.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:494)
        at org.apache.kudu.shaded.com.google.common.net.HostAndPort.getPort(HostAndPort.java:110)
        at org.apache.kudu.util.NetUtil.parseString(NetUtil.java:68)
        at org.apache.kudu.util.NetUtil.parseStrings(NetUtil.java:85)
        at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.<init>(AsyncKuduClient.java:2372)
        at org.apache.kudu.spark.kudu.KuduClientCache$.getAsyncClient(KuduContext.scala:525)
        at org.apache.kudu.spark.kudu.KuduContext.asyncClient$lzycompute(KuduContext.scala:136)
        at org.apache.kudu.spark.kudu.KuduContext.asyncClient(KuduContext.scala:135)
        at org.apache.kudu.spark.kudu.KuduContext.syncClient$lzycompute(KuduContext.scala:133)
        at org.apache.kudu.spark.kudu.KuduContext.syncClient(KuduContext.scala:133)
        at org.apache.kudu.spark.kudu.KuduContext$$anon$1.run(KuduContext.scala:147)
        at org.apache.kudu.spark.kudu.KuduContext$$anon$1.run(KuduContext.scala:145)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:360)
        at org.apache.kudu.spark.kudu.KuduContext.<init>(KuduContext.scala:145)
        at org.apache.kudu.spark.kudu.KuduContext.<init>(KuduContext.scala:67)
        at com.company.new.project.kuduloader.KuduLoader.store(KuduLoader.scala:86)
        at com.company.new.project.kuduloader.Application$$anonfun$main$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Application.scala:42)
        at com.company.new.project.kuduloader.Application$$anonfun$main$1$$anonfun$apply$mcV$sp$1.apply(Application.scala:42)
        at com.company.new.project.kuduloader.Application$$anonfun$main$1$$anonfun$apply$mcV$sp$1.apply(Application.scala:42)
        at com.company.new.project.kuduloader.CoreUtils$.time(CoreUtils.scala:8)
        at com.company.new.project.kuduloader.Application$$anonfun$main$1.apply$mcV$sp(Application.scala:41)
        at com.company.new.project.kuduloader.Application$$anonfun$main$1.apply(Application.scala:27)
        at com.company.new.project.kuduloader.Application$$anonfun$main$1.apply(Application.scala:27)
        at com.company.new.project.kuduloader.CoreUtils$.time(CoreUtils.scala:8)
        at com.company.new.project.kuduloader.Application$.main(Application.scala:27)
        at com.company.new.project.kuduloader.Application.main(Application.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

0 个答案:

没有答案