Question

使用以下版本：

spark 1.6.2
scala 2.10.5
connectorVersion 1.6.8
cassandra_version 3.0.11.1485

在代码中，我正在使用：

RDD.deleteFromCassandra("Keyspace","billing_table")

并获得以下异常：

Exception in thread "main" java.lang.NoSuchMethodError:
   com.datastax.spark.connector.RDDFunctions.deleteFromCassandra$default$3()Lcom/datastax/spark/connector/ColumnSelector;
   at purge.PurgeData$.PurgeSubscriptionsFromClosedAcc(PurgeData.scala:112)
   at purge.PurgeData$.PurgeClosedAccountsFromTables(PurgeData.scala:53)
   at purge.Driver$.main(Driver.scala:29)
   at purge.Driver.main(Driver.scala)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
   at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
   at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
   at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
   at org.apache.spark.deploy.DseSparkSubmitBootstrapper$.main(DseSparkSubmitBootstrapper.scala:48)
   at org.apache.spark.deploy.DseSparkSubmitBootstrapper.main(DseSparkSubmitBootstrapper.scala)

请有人帮忙吗？

谢谢Chandra

Answer 1

看起来您使用spark-submit部署了代码。请记住，您应该在调用spark-submit时添加jar依赖项像这样

./bin/spark-submit \
  --class com.your.package.Class \
  --master local[*] \
  /path/to/your.jar \
  --jars spark-cassandra-connector.jar

或使用sbt assembly plugin创建具有所有必需依赖项的jar。

Answer 2

DeleteFromCassandra仅在SCC版本2.0.0-RC1中引入

https://github.com/datastax/spark-cassandra-connector/commit/6944110c9a2790d28c27bf575b8c065684020f1a

使用＆＃34; - 罐子＆＃34;也是不正确的。将spark-cassandra-connector添加到spark-submit命令，因为这不包括依赖项。而是使用Quick Start Guide

中指定的--packages命令

线程“main”中的异常java.lang.NoSuchMethodError：com.datastax.spark.connector.RDDFunctions.deleteFromCassandra

2 个答案: