Question

在进行弹性搜索连接时出错，我确实谷歌似乎弹性搜索存在问题，有人可以帮忙吗？我从kafka获取数据后使用高音搜索情绪分析我正在使用它并进行情绪分析，然后在json formate中创建弹性搜索索引

I am using below code for build.sbt:

++++++


libraryDependencies ++= Seq("org.apache.spark" %% "spark-core" % "1.6.1",
  "org.apache.spark" %% "spark-mllib" % "1.6.1",
  "org.apache.spark" % "spark-streaming_2.10" % "1.6.1",
  "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.1",
  "org.apache.spark" % "spark-streaming-twitter_2.11" % "1.5.2",
  "org.twitter4j" % "twitter4j-core" % "4.0.4",
  "org.twitter4j" % "twitter4j-stream" % "4.0.4",
  "org.elasticsearch" % "elasticsearch-spark_2.10" % "2.2.0-m1" % "compile",
  "edu.stanford.nlp" % "stanford-corenlp" % "3.5.2" artifacts (Artifact("stanford-corenlp", "models"), Artifact("stanford-corenlp")))


+++++++++++++++++++



    17/04/16 22:37:40 ERROR Executor: Exception in task 1.0 in stage 6.0 (TID 20)

错误登录火花

     org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot find node with id [AieO64NJRQO-vkTU_K56dA] (is HTTP enabled?) from shard [Shard[state=STARTED, primary=true, node=AieO64NJRQO-vkTU_K56dA, name=0, index=twitter2]] in nodes [[{state=STARTED, primary=true, node=AieO64NJRQO-vkTU_K56dA, relocating_node=null, shard=0, index=twitter2, allocation_id={id=0sjxOAtLQr-V9aGCq6xexg}}]]; layout [[[{state=STARTED, primary=true, node=AieO64NJRQO-vkTU_K56dA, relocating_node=null, shard=0, index=twitter2, allocation_id={id=0sjxOAtLQr-V9aGCq6xexg}}], [{state=STARTED, primary=true, node=AieO64NJRQO-vkTU_K56dA, relocating_node=null, shard=1, index=twitter2, `allocation_id={id=9sl-iD3wQaeZyLpSXYhCRg}}], [{state=STARTED, primary=true, node=AieO64NJRQO-vkTU_K56dA, relocating_node=null, shard=2, index=twitter2, allocation_id={id=p4kg_9WWS467ac0KHGRIOQ}}], [{state=STARTED, primary=true, node=AieO64NJRQO-vkTU_K56dA, relocating_node=null, shard=3, index=twitter2, allocation_id={id=DEnyCsYWRZaHDyZexo39UA}}], [{state=STARTED, primary=true, node=AieO64NJRQO-vkTU_K56dA, relocating_node=null, shard=4, index=twitter2, allocation_id={id=Pqn82NKITVSofZrang7sHg}}]]]`

Answer 1

在SBT中使用以下代码将有助于：

// Versions
val sparkVersion = "2.1.1"
val ElasticV = "5.5.2"

libraryDependencies ++= Seq(


  "org.elasticsearch" % "elasticsearch-spark-20_2.11" % "5.5.2" ,
  "org.scala-lang" % "scala-library" % "2.11.11" ,
  "org.apache.spark" % "spark-core_2.11" % "2.1.1",
  "org.apache.spark" % "spark-sql_2.11" % "2.1.1"


)

// Elasticsearch *copies* joda-time code and patch it into their codebase. It causes several issues

    assemblyShadeRules in assembly := Seq(
ShadeRule.rename("org.joda.time.base.**" -> "org.elasticsearch.joda.time.@1").inLibrary("org.elasticsearch" % "elasticsearch" % ElasticV).inProject

    )

在Spark Streamming中与ElasticSearch连接时出错

1 个答案: