Question

尝试从Apache Ignite表加载Spark Dataframe时出现错误。我发现即使在这里有股票“ Person”示例，我也能做到这一点：https://apacheignite-fs.readme.io/docs/ignite-data-frame

我只是在用这些软件包启动的spark shell中进行测试：

index.html

我正在像这样加载DF：

--packages org.apache.ignite:ignite-spark:2.6.0,org.apache.ignite:ignite-core:2.6.0,org.apache.ignite:ignite-scalar:2.6.0,com.datastax.spark:spark-cassandra-connector_2.11:2.3.1

任何数据框操作都会导致package.scala中的失败

scala>  val personDf = spark.read.format(IgniteDataFrameSettings.FORMAT_IGNITE).option(IgniteDataFrameSettings.OPTION_TABLE,  "person").option("config", CONFIG).load()
personDf: org.apache.spark.sql.DataFrame = [AGE: int, NAME: string]

另一个例子：

scala> personDf.show()
java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.expressions.AttributeReference.<init>(Ljava/lang/String;Lorg/apache/spark/sql/types/DataType;ZLorg/apache/spark/sql/types/Metadata;Lorg/apache/spark/sql/catalyst/expressions/ExprId;Lscala/Option;)V
  at org.apache.ignite.spark.impl.optimization.package$.toAttributeReference(package.scala:121)
<snip>

我还尝试了Spark SQL示例：

scala> personDf.count()
java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.expressions.AttributeReference.<init>(Ljava/lang/String;Lorg/apache/spark/sql/types/DataType;ZLorg/apache/spark/sql/types/Metadata;Lorg/apache/spark/sql/catalyst/expressions/ExprId;Lscala/Option;)V
  at org.apache.ignite.spark.impl.optimization.package$.toAttributeReference(package.scala:144)
<snip>

我在某个地方缺少包裹吗？

从Ignite加载Spark数据帧时出现NoSuchMethodError

0 个答案: