Question

以下是我的sbt文件。当我从我的IDE运行时，所有记录都在弹性搜索中被编入索引。但是当我进行组装并将该fatjar提交给spark群集时，它会抛出错误。让我知道组装的fatjar有什么问题。

Caused by: java.lang.NoSuchMethodError: io.netty.buffer.CompositeByteBuf.addComponents(ZLjava/lang/Iterable;)Lio/netty/buffer/CompositeByteBuf;
    at org.elasticsearch.transport.netty4.Netty4Utils.toByteBuf(Netty4Utils.java:78)
    at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:449)
    at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:91)
    at org.elasticsearch.transport.TcpTransport.internalSendMessage(TcpTransport.java:976)
    at org.elasticsearch.transport.TcpTransport.sendRequest(TcpTransport.java:958)
    at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:520)
    at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:465)

我项目的sbt文件

import sbt.Keys._

name := "compute-layer"

version := "1.0"

scalaVersion := "2.10.6"

libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.6.1" excludeAll(ExclusionRule(organization = "org.apache.logging"))
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.6.1" excludeAll(ExclusionRule(organization = "org.apache.logging"))
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.0" excludeAll(ExclusionRule(organization = "org.apache.logging"))
libraryDependencies += "org.elasticsearch.client" % "transport" % "5.1.1" excludeAll(ExclusionRule(organization = "org.apache.logging"))
libraryDependencies += "org.apache.logging.log4j" % "log4j-api" % "2.7"
libraryDependencies += "org.apache.logging.log4j" % "log4j-core" % "2.7"
libraryDependencies += "com.typesafe" % "config" % "1.2.1"
libraryDependencies += "io.netty" % "netty-all" % "4.0.4.Final"

resolvers ++= Seq("clojars" at "https://clojars.org/repo",
  "conjars" at "http://conjars.org/repo",
  "plugins" at "http://repo.spring.io/plugins-release",
  "sonatype" at "http://oss.sonatype.org/content/groups/public/",
  "Online Play Repository" at "http://repo.typesafe.com/typesafe/simple/maven-releases/"
)


assemblyMergeStrategy in assembly := {
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case x => MergeStrategy.first
}


//libraryDependencies += "org.apache.logging.log4j" % "log4j-core" % "2.6.2"

Answer 1

我解决了类似的问题，如下所示：

      <dependency>
        <groupId>org.apache.hbase</groupId>
        <artifactId>hbase-client</artifactId>
        <version>1.3.0</version>
        <exclusions>
            <exclusion>
                <groupId>io.netty</groupId>
                <artifactId>netty-all</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

Answer 2

唯一适用于我的解决方案是将ElasticSearch设置更改为使用netty 3

yourMapSetting.put("transport.type","netty3")
yourMapSetting.put("http.type", "netty3")

Elasticsearch传输客户端：由以下引起：java.lang.NoSuchMethodError：io.netty.buffer.CompositeByteBuf.addComponents

2 个答案: