Question

我有一个半管道，从s3写入pubsub如下

<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css">

<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>

<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"></script>

<style type="text/css">
.btn {
  font-family: 'Cabin', 'Helvetica Neue', Helvetica, Arial, sans-serif;
  font-weight: 400;
  -webkit-transition: all 0.3s ease-in-out;
  -moz-transition: all 0.3s ease-in-out;
  transition: all 0.3s ease-in-out;
  text-transform: uppercase;
  border-radius: 0;
}

.btn.active.btn-default {
  color: #42DCA3;
  border: 1px solid #42DCA3;
  background-color: transparent;
}

.btn.active.btn-default:focus, .btn.active.btn-default:hover {
  color: black;
  border: 1px solid #42DCA3;
  outline: none;
  background-color: #42DCA3;
}
</style>

<section id="gallery" class="content-section text-center">
   <div class="container">
      <div class="row">
        <div class="col-lg-8 mx-auto">
          <button class="btn active btn-default" onclick="filterSelection('all')">Show all</button>
          <button type="button" onclick="filterSelection('sealion')">Sea Lions</button>
        </div>
      </div>
   </div>
</section>

但是，我想同时将记录写入另一个s3存储桶。如何更改上述代码以将其也写入另一个存储桶？

Answer 1

我希望您的问题是关于如何处理输入以便可以将其输出到两个源的问题。下面的想法应该起作用：

PCollection<String> inputCollection= pipeline.apply("Read Text Data", TextIO.read().from(options.getInputFilePattern()));
inputCollection.apply("Write to PubSub",PubsubIO.writeStrings().to(options.getOutputTopic()));
inputCollection.apply("Write to S3 Bucket",<appropriate-IO>.to(options.getOutputPath()));
State p= pipeline.run().waitUntilFinish();

在Apache Beam中写入不同的源

1 个答案: