Question

我是spark / cassandra的相对新手。因此，我有一个基本问题。我编译了一个超级jar并将其加载到我的spark / cassandra服务器。现在我在泡菜中，如何通过cassandra（DSE）环境运行它？我知道spark shell命令是＆＃34; dse spark-submit＆＃34;但当我尝试做一个＆＃34; dse spark-submit＆＃34;我得到了一个＆＃34; NullPointerException＆＃34;

以下是完整输出：

Exception in thread "main" java.lang.NullPointerException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

程序代码非常基础，并且已经证明可以在spark shell中使用包xxx.seaoxxxx

import com.datastax.spark.connector._
import org.apache.spark.{SparkConf, SparkContext}


class test {
  def main(args: Array[String]){
    val conf = new SparkConf(true).set("spark.cassandra.connection.host", "xx.xxx.xx.xx")
      .setAppName("Seasonality")

    val sc = new SparkContext("spark://xx.xxx.xx.xx:7077", "Season", conf)

    val ks = "loadset"
    val incf =  "period"

    val rdd = sc.cassandraTable(ks, incf)
    rdd.count
    println("done with test")
    sc.stop()
  }
}

spark-submit代码如下： dse spark-submit \ --class xxx.seaoxxxx.test \ --master spark：//xxx.xx.x.xxx：7077 \ /home/ubuntu/spark/Seasonality_v6-assembly-1.0.1.jar 100

谢谢，

埃里克

Answer 1

当前版本DataStax Enterprise 4.5支持dse spark-class而不是dse spark-submit：http://www.datastax.com/documentation/datastax_enterprise/4.5/datastax_enterprise/spark/sparkStart.html?scroll=sparkStart__spkShrkLaunch

Cassandra Spark职位提交

1 个答案: