Question

我在Java中使用Spark流。我将sparkconfig obj配置为SparkConf sparkConf = new SparkConf().setAppName("MyApp").setMaster("local[2]") .set("spark.streaming.stopGracefullyOnShutdown","true") .set("redis.host", "localhost") .set("redis.port", "6379"); 并在JavastreamingContext中传递配置obj。

JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.milliseconds(1000));

我如何使用jssc对象访问redis。预先感谢。

Answer 1

这将从Redis列表中创建一个流

  SparkConf sparkConf = new SparkConf().setAppName("MyApp").setMaster("local[2]")
            .set("spark.streaming.stopGracefullyOnShutdown", "true")
            .set("redis.host", "localhost")
            .set("redis.port", "6379");


    JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.milliseconds(1000));

    RedisConfig redisConfig = new RedisConfig(new RedisEndpoint(sparkConf));

    RedisStreamingContext redisStreamingContext = new RedisStreamingContext(jssc.ssc());
    String[] keys = new String[]{"myList"};
    RedisInputDStream<Tuple2<String, String>> redisStream =
            redisStreamingContext.createRedisStream(keys, StorageLevel.MEMORY_ONLY(), redisConfig);

    redisStream.print();

    jssc.start();
    jssc.awaitTermination();

将一些数据推送到列表中

LPUSH "myList" "aaaa"
LPUSH "myList" "bbbb"

如何在Spark Streaming中使用Redis？

1 个答案: