Question

我想创建一个包含2个不同主题的2个kafkaSpouts的拓扑，并根据bolt中的sourceComponent将这两个spout合并为一个流。

std::tuple<typename std::decay<Args>::type...> para;
//                  ~~~~~~~~~^

}

在Bolt执行方法中我正在做

public class Topology {
private static final String topic1 = Real2
private static final String topic2 = Real1


public static void main(String[] args) throws AlreadyAliveException,
        InvalidTopologyException, IOException {

    BasicConfigurator.configure();

    String zookeeper_root = "";
    SpoutConfig kafkaConfig1 = new SpoutConfig(localhost:2181, topic1,
            zookeeper_root, "Real1KafkaSpout");

    SpoutConfig kafkaConfig2 = new SpoutConfig(localhost:2181, topic2,
            zookeeper_root, "Real2KafkaSpout");


    kafkaConfigRealTime.scheme = new SchemeAsMultiScheme(new StringScheme());
    kafkaConfigRealTime.forceFromStart = true;

    kafkaConfigHistorical.scheme = new SchemeAsMultiScheme(
            new StringScheme());
    kafkaConfigHistorical.forceFromStart = true;



    TopologyBuilder builder = new TopologyBuilder();

    builder.setSpout("Real1", new KafkaSpout(
            kafkaConfig1), 2);
    builder.setSpout("Real2", new KafkaSpout(
            kafkaConfig2), 2);

    builder.setBolt("StreamMerging", new StreamMergingBolt(), 2)
            .setNumTasks(2).shuffleGrouping("Real1")
            .shuffleGrouping("Real2");



    Config config = new Config();
    config.put("hdfs.config", yamlConf);

    config.setDebug(false);
    config.setMaxSpoutPending(10000);

    if (args.length == 0) {
        LocalCluster cluster = new LocalCluster();

        cluster.submitTopology("Topology", config,
                builder.createTopology());
        cluster.killTopology("Topology");
        cluster.shutdown();
    } else {

        StormSubmitter.submitTopology(args[0], config,
                builder.createTopology());
    }

    try {
        Thread.sleep(6000);
    } catch (InterruptedException ex) {
        ex.printStackTrace();
    }

}

所以我想存储来自每个流的单独文件元组那就是我想将Real1KafkaSpout的元组存储到file1，将Real2KafkaSpout存储到file2。我怎么能这样做我在这一点上受到了打击

Answer 1

你可以这样做：

public void execute(Tuple input, BasicOutputCollector collector) {
    String id = input.getSourceComponent();
    if(id.equals("Real1")) {
        // store into file1
    } else {
        assert (id.equals("Real2");
        // store in file2
    }
}

您可以在Bolt.open(...)中打开这两个文件。

但是，我想知道为什么要使用单个拓扑来实现这一点。如果你只将Kafka源码中的数据写入文件1，将数据从Kafka源码2写入文件2，你可以简单地创建两个拓扑...（当然，你只编程一次，并且只为两种情况配置不同）

Answer 2

当我通过以下代码执行此操作时的有线结果

public void execute(Tuple input, BasicOutputCollector collector) {
String id = input.getSourceComponent();
if(id.equals("Real1")) {
   String json = input.getString(0);
  //writetoHDFS1(json)
   } else {
      assert (id.equals("Real2");
    String json = input.getString(0);
 //writetoHDFS2(json)
  }
 }

2个喷出一个风暴螺栓

2 个答案: