Question

我打算使用apache flink使用flink将读/写数据写入cassandra。我希望使用flink-connector-cassandra，但我找不到连接器的好文档/示例。

请您指出使用Apache Flink从cassandra读取和写入数据的正确方法。我只看到了纯粹用于写入的接收器示例？ apache flink是否也意味着从cassandra读取数据，类似于apache spark？

Answer 1

我有同样的问题，这就是我在寻找的东西。我不知道它是否已经过度简化了你需要的东西，但我认为我应该表明它。

ClusterBuilder cb = new ClusterBuilder() {
        @Override
        public Cluster buildCluster(Cluster.Builder builder) {
            return builder.addContactPoint("urlToUse.com").withPort(9042).build();
        }
    };

    CassandraInputFormat<Tuple2<String, String>> cassandraInputFormat = new CassandraInputFormat<>("SELECT * FROM example.cassandraconnectorexample", cb);

    cassandraInputFormat.configure(null);
    cassandraInputFormat.open(null);

    Tuple2<String, String> testOutputTuple = new Tuple2<>();
    cassandraInputFormat.nextRecord(testOutputTuple);

    System.out.println("column1: " + testOutputTuple.f0);
    System.out.println("column2: " + testOutputTuple.f1);

我想出这个的方式是因为找到了“CassandraInputFormat”类的代码并看到它是如何工作的（http://www.javatips.net/api/flink-master/flink-connectors/flink-connector-cassandra/src/main/java/org/apache/flink/batch/connectors/cassandra/CassandraInputFormat.java）。老实说，我希望它只是一种格式，而不是基于名字的Cassandra的全班阅读，我感觉其他人可能会想到同样的事情。

Answer 2

您可以使用RichFlatMapFunction扩展课程

class MongoMapper extends RichFlatMapFunction[JsonNode,JsonNode]{
  var userCollection: MongoCollection[Document] = _
  override def open(parameters: Configuration): Unit = {
// do something here like opening connection
    val client: MongoClient = MongoClient("mongodb://localhost:10000")

    userCollection = client.getDatabase("gp_stage").getCollection("users").withReadPreference(ReadPreference.secondaryPreferred())
    super.open(parameters)
  }
  override def flatMap(event: JsonNode, out: Collector[JsonNode]): Unit = {

// Do something here per record and this function can make use of objects initialized via open
      userCollection.find(Filters.eq("_id", somevalue)).limit(1).first().subscribe(
        (result: Document) => {
//          println(result)
                      },
      (t: Throwable) =>{
        println(t)
      },
        ()=>{
          out.collect(event)
        }
      )
    }


  }

}

基本上open函数每个worker执行一次，flatmap每个记录执行一次。该示例适用于mongo，但可以类似地用于cassandra

Answer 3

根据我的理解，您理解管道的第一步是从Cassandra读取数据而不是编写RichFlatMapFunction，您应该编写自己的RichSourceFunction

作为参考，您可以查看WikipediaEditsSource的简单实现。

Answer 4

    ClusterBuilder cb = new ClusterBuilder() {
        @Override
        public Cluster buildCluster(Cluster.Builder builder) {
            return builder.addContactPoint("localhost").withPort(9042).build();
        }
    };

    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
    InputFormat inputFormat = new CassandraInputFormat<Tuple3<Integer, Integer, Integer>>("SELECT * FROM test.example;", cb);//, TypeInformation.of(Tuple3.class));

    DataStreamSource t = env.createInput(inputFormat,  TupleTypeInfo.of(new TypeHint<Tuple3<Integer, Integer,Integer>>() {}));
    tableEnv.registerDataStream("t1",t);
    Table t2 = tableEnv.sql("select * from t1");

    t2.printSchema();

阅读＆amp;使用apache flink Java API将数据写入cassandra

4 个答案: