Question

我正在使用Spark 2.1.1 Scala 2.11.8和Java 8

主Java Spark类：

import java.util.Arrays;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
import java.util.regex.Pattern;

import org.apache.spark.SparkConf;
import org.apache.spark.streaming.Durations;
import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaPairDStream;
import org.apache.spark.streaming.api.java.JavaPairInputDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
import org.apache.spark.streaming.kafka.KafkaUtils;

import kafka.serializer.StringDecoder;
import scala.Tuple2;

public class SparkSample {

     private static final Pattern SPACE = Pattern.compile(" ");

    public static void main(String[] args) throws Exception{

        SparkConf sparkConf = new SparkConf().setAppName("App");

        //Duration the interval at which streaming data will be divided into batches
        JavaStreamingContext javaStreamingContext = new     JavaStreamingContext(sparkConf, Durations.seconds(10));

        Set<String> topicsSet = new HashSet<>(Arrays.asList("MY-TOPIC".split(",")));
        Map<String, String> kafkaConfiguration = new HashMap<>();
        kafkaConfiguration.put("metadata.broker.list", "MYIP:9092");
        kafkaConfiguration.put("group.id", "Stream");

        JavaPairInputDStream<String, String> messages = KafkaUtils.createDirectStream(
            javaStreamingContext,
            String.class,
            String.class,
            StringDecoder.class,
            StringDecoder.class,
            kafkaConfiguration,
            topicsSet
    );
        messages.print();
        //BELOW PART THROWS ERRORS IF UNCOMMENTED
        //JavaDStream<String> lines = messages.map(Tuple2::_2);
        //JavaDStream<String> words = lines.flatMap(x ->     Arrays.asList(SPACE.split(x)).iterator());
        //JavaPairDStream<String, Integer> wordCounts = words.mapToPair(s -> new Tuple2<>(s, 1))
        //    .reduceByKey((i1, i2) -> i1 + i2);
        //wordCounts.print();

        // Start the computation
        javaStreamingContext.start();
        javaStreamingContext.awaitTermination();
    }

}

依赖关系：

  <dependencies>
<dependency>
  <groupId>junit</groupId>
  <artifactId>junit</artifactId>
  <version>4.12</version>
  <scope>test</scope>
</dependency>
 <dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-core_2.11</artifactId>
  <version>2.1.1</version>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-kafka_2.11</artifactId> <!-- spark-streaming-kafka_2.11 spark-streaming-kafka-0-8_2.11-->
    <version>1.6.3</version> <!-- 1.6.3 2.0.1-->
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming_2.11</artifactId>
    <version>2.1.1</version>
</dependency>
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>0.8.2.2</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka_2.11</artifactId>
    <version>0.8.2.2</version>
</dependency>

注释的代码在取消注释时会抛出以下错误：

Unsupported major.minor version 52.0

如果我对它进行评论，那么它会打印好的消息。有什么想法吗？我正在使用Java 8构建/编译。

Answer 1

该错误表明您正在运行为针对运行它的运行时版本而言太新的版本编译的Java代码。 Java 7或更早版本可能是您机器上的默认设置（驱动程序，工作程序，主程序等）

您需要检查和/或修复的事项：

检查运行时是否与目标匹配。在运行spark应用程序的命令行上使用java -version。 必须在驱动程序计算机和群集中的每台计算机上执行此操作。所有这些都必须为您的代码提供正确的Java版本。

如果您要为早期Java版本进行编译（如果您想在早期版本上运行），则可能需要更改目标编译版本：

<plugin>
  <artifactId>maven-compiler-plugin</artifactId>
  <version>...</version>
  <configuration>
      <source>1.7</source>
      <target>1.7</target>

确保您的spark用户JAVA_HOME解析为正确的Java安装目录。另请检查PATH环境变量是否包含正确的$JAVA_HOME/bin路径。

Spark Java版本错误

1 个答案: