Question

我正在构建一个应用程序，该应用程序从redis中的列表中读取json元素，并使用spark流式传输它们。这是我写的：

public void readTheStream() throws UnknownHostException, IOException {
        SparkConf sparkConf = new SparkConf().setMaster("local[*]").setAppName("Merge").set("redis.host", "localhost")
                .set("redis.port", "6379");;

        JavaSparkContext ctx = JavaSparkContext.fromSparkContext(SparkContext.getOrCreate(sparkConf));
        JavaStreamingContext context = new JavaStreamingContext(ctx, Durations.seconds(1));
}

我如何使用jssc对象访问redis。预先感谢。

Answer 1

下面是一个示例，该示例读取myList并将列表项打印到控制台：

SparkConf sparkConf = new SparkConf().setAppName("MyApp").setMaster("local[*]")
                .set("redis.host", "localhost")
                .set("redis.port", "6379");


JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.milliseconds(1000));

RedisConfig redisConfig = new RedisConfig(new RedisEndpoint(sparkConf));

RedisStreamingContext redisStreamingContext = new RedisStreamingContext(jssc.ssc());
String[] keys = new String[]{"myList"};
RedisInputDStream<Tuple2<String, String>> redisStream =
        redisStreamingContext.createRedisStream(keys, StorageLevel.MEMORY_ONLY(), redisConfig);

redisStream.print();

jssc.start();
jssc.awaitTermination();

如何在Spark Streaming中使用Redis

1 个答案: