我尝试使用Dataflow从包含大约20亿行的表中执行行处理,但是从BigQuery读取时我的作业失败,并显示以下错误消息:
error: Resources exceeded during query execution.
我的管道的阅读部分看起来像这样:
DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class);
options.setProject(project);
options.setStagingLocation(staging_location);
options.setRunner(DataflowPipelineRunner.class);
Pipeline p = Pipeline.create(options);
PCollection<TableRow> rows = p.apply(BigQueryIO.Read.fromQuery(query))
是否可以使用BigQueryIO运行BigQuery查询并允许大结果?