Question

我的Spark scala代码如下：

val input = sc.newAPIHadoopRDD(jconf, classOf[CqlInputFormat], classOf[LongWritable], classOf[Row])

CqlInputFormat类在Cassandra的源代码中实现。我试图将其转换为Java代码，并且可以正常工作。但是它无法使用Scala代码进行构建。

inferred type arguments[org.apache.hadoop.io.LongWritable,com.datastax.driver.core.Row,org.apache.cassandra.hadoop.cql3.CqlInputFormat] do not conform to method newAPIHadoopRDD's type parameter bounds [K,V,F <: org.apache.hadoop.mapreduce.InputFormat[K,V]]
[error]         val input = sc.newAPIHadoopRDD(jconf, classOf[CqlInputFormat], classOf[LongWritable], classOf[Row]);

[error] /home/project/past/experiments/query/SparkApp/src/main/scala/SparkReader.scala:46: type mismatch;
[error]  found   : Class[org.apache.cassandra.hadoop.cql3.CqlInputFormat](classOf[org.apache.cassandra.hadoop.cql3.CqlInputFormat])
[error]  required: Class[F]
[error]         val input = sc.newAPIHadoopRDD(jconf, classOf[CqlInputFormat], classOf[LongWritable], classOf[Row]);
[error]                                                      ^
[error] /home/project/past/experiments/query/SparkApp/src/main/scala/SparkReader.scala:46: type mismatch;
[error]  found   : Class[org.apache.hadoop.io.LongWritable](classOf[org.apache.hadoop.io.LongWritable])
[error]  required: Class[K]
[error]         val input = sc.newAPIHadoopRDD(jconf, classOf[CqlInputFormat], classOf[LongWritable], classOf[Row]);
[error]                                                                               

[error] /home/project/past/experiments/query/SparkApp/src/main/scala/SparkReader.scala:46: type mismatch;
[error]  found   : Class[com.datastax.driver.core.Row](classOf[com.datastax.driver.core.Row])
[error]  required: Class[V]
[error]  val input = sc.newAPIHadoopRDD(jconf, classOf[CqlInputFormat], classOf[LongWritable], classOf[Row]);
[error]                                                                                                      
[error] four errors found
[error] (compile:compileIncremental) Compilation failed

有什么建议吗？谢谢。

Answer 1

如果您使用的是Spark，则需要使用Spark Cassandra Connector而不是使用Hadoop集成。而且最好使用DataFrames ...

我建议您使用DS320 course来详细了解Spark + Cassandra。

Cassandra的CqlInputFormat无法在Scala中构建，但适用于Java

1 个答案: