Question

我想使用RDD[(Object,Object)] messages将saveToEs保存到ElasticSearch中。在下面的代码中，我迭代DStream[String] transformed，对于每个RDD[String] rdd，我使用RDD[(Object,Object)]创建prepare。问题是saveToEs标记为红色，但我将工件elasticsearch-hadoop（版本2.3.2）添加到pom.xml中：

transformed.foreachRDD(rdd => {
  if (!rdd.isEmpty) {
    val messages = rdd.map(prepare)
    messages.saveToEs(ec.getResource().toString)
  }
})

private def prepare(message:String):(Object,Object) = {

   val m = JSON.parseFull(message) match {
      case Some(map) => map.asInstanceOf[Map[String,String]]
      case None => Map.empty[String,String]
   }

   val kw = NullWritable.get

   val vw = new MapWritable
   for ((k, v) <- m) vw.put(new Text(k), new Text(v))

   (kw, vw)       
}

Answer 1

您需要导入正确的包以使用 <dependency> <groupId>org.elasticsearch</groupId> <artifactId>elasticsearch-spark_2.10</artifactId> <version>2.2.0</version> </dependency>

中定义的saveToES方法

 [Symfony\Component\Config\Exception\FileLoaderLoadException]
  Warning: trim() expects parameter 1 to be string, array given in C:/xampp-server/htdocs/sarkamartinu/app/config\con
  fig.yml (which is being imported from "C:/xampp-server/htdocs/sarkamartinu/app/config/config_dev.yml").

来源：

https://www.elastic.co/guide/en/elasticsearch/hadoop/master/spark.html#spark-write-scala

Spark Streaming + Elasticsearch：无法解析符号saveToEs

1 个答案: