使用spark-submit提交的SparkR作业(R脚本)在BigInsights Hadoop集群中失败

时间:2016-08-07 18:08:29

标签: r ibm-cloud apache-spark-sql cloudant biginsights

我创建了IBM BigInsights服务,其中包含5个节点的hadoop集群(包括带SparkR的Apache Spark)。我尝试使用SparkR连接cloudant db并获取一些数据并进行一些处理。

使用spark-submit提交的SparkR作业(R脚本)在BigInsights Hadoop集群中失败。 我创建了SparkR脚本并运行了以下代码,

-bash-4.1$ spark-submit --master local[2] test_sparkr.R
16/08/07 17:43:40 WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead.
Error: could not find function "sparkR.init"
Execution halted
-bash-4.1$

test_sparkr.R文件的内容是:

# Creating SparkConext and connecting to Cloudant DB
sc <- sparkR.init(sparkEnv = list("cloudant.host"="<<cloudant-host-name>>","<<><<cloudant-user-name>>>","cloudant.password"="<<cloudant-password>>", "jsonstore.rdd.schemaSampleSize"="-1"))

# Database to be connected to extract the data
database <- "testdata"
# Creating Spark SQL Context
sqlContext <- sparkRSQL.init(sc)
# Creating DataFrame for the "testdata" Cloudant DB
testDataDF <- read.df(sqlContext, database, header='true', source = "com.cloudant.spark",inferSchema='true')

如何在IBM BigInsights中安装spark-cloudant连接器并解决问题。请做必要的。非常感谢帮助。

1 个答案:

答案 0 :(得分:0)

我相信火花云量连接器还不适合R.

希望我能在这个时候更新这个答案!