Question

我正在使用简单的Kafka读取器和国家/地区密钥，在运行spark时出现以下错误

线程“ main”中的异常java.lang.IllegalStateException：没有为未绑定转换注册的TransformEvaluator读取（KafkaUnboundedSource）在org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState（Preconditions.java:588）在org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator $ Translator.translateUnbounded（StreamingTransformTranslator.java:560）处在org.apache.beam.runners.spark.SparkRunner $ Evaluator.translate（SparkRunner.java:451）

我无法获得有关此错误的任何信息

 pipeline
                        .apply(
                                KafkaIO.<String, String>read()
                                        .withBootstrapServers(options.getBootstrap())
                                        .withTopic(options.getInputTopic())
                                        .withKeyDeserializer(StringDeserializer.class)
                                        .withValueDeserializer(StringDeserializer.class)
                                        .withConsumerConfigUpdates(ImmutableMap.of("group.id", (Object) "test1"))
                                        .withoutMetadata())


                        .apply(ParDo.of(new DoFn<KV<String, String>, MetricData>() {....

Answer 1

我遇到了同样的异常，对我来说（请注意：我使用Apache Beam和sbt），添加以下程序集合并策略可以解决此问题：

assemblyMergeStrategy in assembly ~= { old =>
    {
      case s if s.endsWith(".properties")           => MergeStrategy.filterDistinctLines
      case s if s.endsWith("pom.xml")               => MergeStrategy.last
      case s if s.endsWith(".class")                => MergeStrategy.last
      case s if s.endsWith(".proto")                => MergeStrategy.last
      case s if s.endsWith("libjansi.jnilib")       => MergeStrategy.last
      case s if s.endsWith("jansi.dll")             => MergeStrategy.rename
      case s if s.endsWith("libjansi.so")           => MergeStrategy.rename
      case s if s.endsWith("libsnappyjava.jnilib")  => MergeStrategy.last
      case s if s.endsWith("libsnappyjava.so")      => MergeStrategy.last
      case s if s.endsWith("snappyjava_snappy.dll") => MergeStrategy.last
      case s if s.endsWith(".dtd")                  => MergeStrategy.rename
      case s if s.endsWith(".xsd")                  => MergeStrategy.rename
      case PathList("META-INF", "services", "org.apache.hadoop.fs.FileSystem") =>
        MergeStrategy.filterDistinctLines
      case s => old(s)
    }

因此，您可能希望查看官方documentation中的“阴影”部分，并将以下内容添加到您的pom.xml文件中（我想它还不存在。请原谅。我错了。）：

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-shade-plugin</artifactId>
  <configuration>
    <createDependencyReducedPom>false</createDependencyReducedPom>
    <filters>
      <filter>
        <artifact>*:*</artifact>
        <excludes>
          <exclude>META-INF/*.SF</exclude>
          <exclude>META-INF/*.DSA</exclude>
          <exclude>META-INF/*.RSA</exclude>
        </excludes>
      </filter>
    </filters>
  </configuration>
  <executions>
    <execution>
      <phase>package</phase>
      <goals>
        <goal>shade</goal>
      </goals>
      <configuration>
        <shadedArtifactAttached>true</shadedArtifactAttached>
        <shadedClassifierName>shaded</shadedClassifierName>
        <transformers>
          <transformer
            implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
        </transformers>
      </configuration>
    </execution>
  </executions>
</plugin>

Apache beam Kafka [没有为未绑定转换读（KafkaUnboundedSource）注册任何TransformEvaluator]

1 个答案: