我正在尝试在Apache Kudu上运行我的代码。虽然在我的本地kudu设置上运行正常,但我在同一台计算机上拥有主服务器和平板电脑服务器。现在,通过在Cloudera Hadoop env上通过rpm软件包安装手动安装Kudu,我得到了例外。
我正在尝试通过spark加载一个csv文件并坚持到Kudu。我已经在工作状态下设置了以下内容:
在不同的节点上安装了Kudu主机和平板电脑(它们正在运行) 在所有群集节点上都设置了新用户(kudu)的kereberos票证。 (但是,只有一台母版和一台平板电脑,只需对其中两个进行初始化即可)。针对每个组件(例如spark,hive等)都制定了护林员政策,以便向另一个组件/从另一个组件提交作业
/**
* Storing the dataframe into Kudu
*
* @param df the dataframe to store.
*/
def store(df: DataFrame): Unit = {
// Init configuration.
log.info("kuduMaster: " + config.kuduMaster)
val kuduContext = new KuduContext(config.kuduMaster, config.sparkSession.sparkContext)
val primaryKeys = SchemaMapper.primaryKeys(config.tableName).get
// Create Table if non existing
if (!kuduContext.tableExists(config.tableName)) {
log.info("Create Table, tableName: " + config.tableName)
kuduContext.createTable(
config.tableName, df.schema, primaryKeys, new CreateTableOptions()
.setNumReplicas(config.kuduNumReplicas)
.addHashPartitions(primaryKeys.asJava, config.kuduBuckets))
}
Spark插入Kudu时出现异常。成功启动Kerberos主体后: 以下是每次出现的例外情况。我试图从kudu安装所在的Hadoop的同一节点上并通过kudu用户运行它,但是它不起作用。
19/09/10 18:33:43 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.
19/09/10 18:33:43 INFO CoreUtils$: Elapsed time [KuduLoader load csv->df]: 1.727817664s
19/09/10 18:33:43 INFO KuduLoader$: kuduMaster: kudu://mynode.ic.company.net:7051
19/09/10 18:33:43 INFO KuduContext: Logging in as principal kudu@mycluster.IC.COMPANY.NET with keytab /home/kudu/kudu.keytab
Exception in thread "main" java.lang.IllegalStateException
at org.apache.kudu.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:494)
at org.apache.kudu.shaded.com.google.common.net.HostAndPort.getPort(HostAndPort.java:110)
at org.apache.kudu.util.NetUtil.parseString(NetUtil.java:68)
at org.apache.kudu.util.NetUtil.parseStrings(NetUtil.java:85)
at org.apache.kudu.client.AsyncKuduClient$AsyncKuduClientBuilder.<init>(AsyncKuduClient.java:2372)
at org.apache.kudu.spark.kudu.KuduClientCache$.getAsyncClient(KuduContext.scala:525)
at org.apache.kudu.spark.kudu.KuduContext.asyncClient$lzycompute(KuduContext.scala:136)
at org.apache.kudu.spark.kudu.KuduContext.asyncClient(KuduContext.scala:135)
at org.apache.kudu.spark.kudu.KuduContext.syncClient$lzycompute(KuduContext.scala:133)
at org.apache.kudu.spark.kudu.KuduContext.syncClient(KuduContext.scala:133)
at org.apache.kudu.spark.kudu.KuduContext$$anon$1.run(KuduContext.scala:147)
at org.apache.kudu.spark.kudu.KuduContext$$anon$1.run(KuduContext.scala:145)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.kudu.spark.kudu.KuduContext.<init>(KuduContext.scala:145)
at org.apache.kudu.spark.kudu.KuduContext.<init>(KuduContext.scala:67)
at com.company.new.project.kuduloader.KuduLoader.store(KuduLoader.scala:86)
at com.company.new.project.kuduloader.Application$$anonfun$main$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Application.scala:42)
at com.company.new.project.kuduloader.Application$$anonfun$main$1$$anonfun$apply$mcV$sp$1.apply(Application.scala:42)
at com.company.new.project.kuduloader.Application$$anonfun$main$1$$anonfun$apply$mcV$sp$1.apply(Application.scala:42)
at com.company.new.project.kuduloader.CoreUtils$.time(CoreUtils.scala:8)
at com.company.new.project.kuduloader.Application$$anonfun$main$1.apply$mcV$sp(Application.scala:41)
at com.company.new.project.kuduloader.Application$$anonfun$main$1.apply(Application.scala:27)
at com.company.new.project.kuduloader.Application$$anonfun$main$1.apply(Application.scala:27)
at com.company.new.project.kuduloader.CoreUtils$.time(CoreUtils.scala:8)
at com.company.new.project.kuduloader.Application$.main(Application.scala:27)
at com.company.new.project.kuduloader.Application.main(Application.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)