Question

我正在尝试传递BigTable tableId，instanceId和projectId，它们在执行时在TemplateOption类中定义为ValueProvider，因为它们是运行时值，但是它们没有得到新值的尊重。使用在构造管道时定义的旧值执行pipleine。我应该做些什么改变才能在运行时尊重价值观？

Pipeline p = Pipeline.create(options);
com.google.cloud.bigtable.config.BigtableOptions.Builder optionsBuilder =
        new com.google.cloud.bigtable.config.BigtableOptions.Builder().
                setProjectId("my-project");   

PCollection<com.google.bigtable.v2.Row> row = p.apply("filtered read", org.apache.beam.sdk.io.gcp.bigtable.BigtableIO.read().withBigtableOptions(optionsBuilder).withoutValidation().withInstanceId(options.getInstanceId()).withProjectId(options.getProjectId()).withTableId(options.getTableId()));
PCollection<KV<Integer,String>> convertToKV = row.apply(ParDo.of(new ConvertToKV()));

我的Option类如下： -

@Default.String("my-project")
@Description("The Google Cloud project ID for the Cloud Bigtable instance.")
ValueProvider<String> getProjectId();
void setProjectId(ValueProvider<String> projectId);

@Default.String("my-instance")
@Description("The Google Cloud Bigtable instance ID .")
ValueProvider<String> getInstanceId();
void setInstanceId(ValueProvider<String> instanceId);

@Default.String("my-test")
@Description("The Cloud Bigtable table ID in the instance." )
ValueProvider<String> getTableId();
void setTableId(ValueProvider<String> tableId);

@Description("bucket name")
@Default.String("mybucket")
ValueProvider<String> getBucketName();
void setBucketName(ValueProvider<String> bucketName);

任何帮助都会非常感激。

Answer 1

我确实认为在施工时验证运行时参数是个问题。但是，我不理解的是不遵守使用模板执行管道时传递的运行时参数。

如何传递运行时参数？它应该是这样的：

var model: RecordedVideoModel! {
    didSet {

        //Customize
        let blurEffect = UIBlurEffect(style: UIBlurEffectStyle.light)
        let blurEffectView = UIVisualEffectView(effect: blurEffect)
        blurEffectView.frame = (self.blurView?.bounds)!
        blurEffectView.autoresizingMask = [.flexibleWidth, .flexibleHeight]
        blurView.addSubview(blurEffectView)

        //VideoLabel and Text
        self.videoLabel?.textColor = UIColor.white
        self.videoLabel?.font = UIFont.init(name: FontTypeBold, size: lightTextSize)
        self.videoLabel?.text = model.username

         //Duration/TimeStamp and Text
        self.durationLabel?.textColor = UIColor.white
        self.durationLabel?.font = UIFont.init(name: FontTypeRegular, size: lightTextSize)
        let now = Date()
        self.durationLabel.text = String(format:"%@ ago",now.offset(from: DateConverter.getDateFromString(dateToConvert: model.session_timestamp!)))

        let contentUrl = URL(string: String(format:"%@",model.video_cdn_url!))
        player = AVPlayer(url: contentUrl!)
        playerLayer = AVPlayerLayer.init(player: player)
        playerLayer.frame = (self.videoView?.bounds)!
        self.playerLayer?.backgroundColor = UIColor.clear.cgColor
        playerLayer.videoGravity = AVLayerVideoGravity.resizeAspectFill
        self.videoView?.layer.addSublayer(playerLayer)
        player?.isMuted = true
        player!.addPeriodicTimeObserver(forInterval: CMTimeMakeWithSeconds(1, 1), queue: DispatchQueue.main) { (progressTime) -> Void in
            let duration = CMTimeGetSeconds(progressTime)
            let seconds = Int(duration .truncatingRemainder(dividingBy:  60))
            if seconds == 5 {
                self.player?.seek(to: kCMTimeZero)
            }

        }
        //player?.play()
        player?.playImmediately(atRate: 1.0)
    }
}
override func prepareForReuse()
 {
    super.prepareForReuse()
  //self.player = nil
    self.playerLayer.removeFromSuperlayer()
    self.player?.pause()
    for view in self.blurView.subviews
    {
        view.removeFromSuperview()
    }

}

参见＆＃34;创建模板＆＃34;有关详细信息：https://cloud.google.com/dataflow/docs/templates/creating-templates

构建模板后，您可以使用运行时参数执行管道。例如：

  public interface WordCountOptions extends PipelineOptions {
    @Description("Path of the file to read from")
    @Default.String("gs://dataflow-samples/shakespeare/kinglear.txt")
    ValueProvider<String> getInputFile();
    void setInputFile(ValueProvider<String> value);
  }

    public static void main(String[] args) {
        WordCountOptions options =
              PipelineOptionsFactory.fromArgs(args).withValidation()
                .as(WordCountOptions.class);
        Pipeline p = Pipeline.create(options);

参见＆＃34;执行模板＆＃34;有关详细信息：https://cloud.google.com/dataflow/docs/templates/executing-templates

注意：如果在执行管道时没有传递运行时参数，则参数将具有默认值或null。

希望这有帮助！

Answer 2

我相信在创建模板时，--inputFiles会与模板捆绑在一起。

请参阅注释1：“除模板文件外，模板化管道执行还依赖于在模板创建时暂存和引用的文件。如果移动或删除了暂存文件，则管道执行将失败。“

此线程似乎也与2

ValueProvider类型参数未在模板执行时

3 个答案: