Question

通过引用this链接，我尝试在spark Dataframe中查询cassandra表

val spark = SparkSession
          .builder()
          .appName("CassandraSpark")
          .config("spark.cassandra.connection.host", "127.0.0.1")
          .config("spark.cassandra.connection.port", "9042")
          .master("local[2]")
          .getOrCreate();

我正在使用的节点是SearchAnalytics节点使用这个spark会话，我尝试了sql查询

val ss = spark.sql("select * from killr_video.videos where solr_query = '{\"q\":\"video_id:1\"}'")

该表已启用搜索索引。

运行程序后，这是我得到的错误

Exception in thread "main" org.apache.spark.sql.AnalysisException: Table or view not found: `killr_video`.`videos`; line 1 pos 14;

'项目[*] + - 'UnresolvedRelation killr_video。videos

at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:82)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:78)
at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:126)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:78)
at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:91)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:52)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:66)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:623)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:691)

如何将Cassandra数据导入Spark？

Answer 1

从此错误消息中，您似乎使用独立Spark运行代码，而不是通过DSE Analytics（通过dse spark-submit或dse spark）。

在这种情况下，您需要使用dse client-tool＆amp;注册表 - DSE documentation describes如何为所有表格执行此操作。 spark-sql：

dse client-tool --use-server-config spark sql-schema --all > output.sql
spark-sql --jars byos-5.1.jar -f  output.sql

对于我的例子，它看起来如下：

USE test;
CREATE TABLE t122
       USING org.apache.spark.sql.cassandra
       OPTIONS (
                keyspace "test",
                table "t122",
                pushdown "true");

以下是solr_query的一个示例，如果我在以dse spark开头的spark-shell中运行它，那么它只能在盒子外工作：

scala> val ss = spark.sql("select * from test.t122 where solr_query='{\"q\":\"t:t2\"}'").show
+---+----------+---+
| id|solr_query|  t|
+---+----------+---+
|  2|      null| t2|
+---+----------+---+

为了让您的生活更轻松，最好使用DSE Analytics，而不是bring your own spark。

带有Scala错误的DSE SearchAnalytics

1 个答案: