Question

我可以使用DirectRunner运行此代码并且工作正常。 DataflowRunner与apache-beam-sdk==0.6.0.dev0崩溃：

TypeError：process（）只需要4个参数（给定3个）[while 运行'write_text / Write / WriteImpl / WriteBundles']`

我的apache-beam-sdk是按照指示中的指示从主人那里克隆和构建的。它构建为NewDoFn。我对该版本持怀疑态度，因为（我认为）我看到代码更改而最近没有版本更改（DirectRunner消失但版本没有改变）。

我不确定它是否是问题的根源，但似乎安装的sdk和数据流容器不匹配。我得到了另一个不匹配类型错误，其中element将DoFn.process()直接传递到DataflowRunner，而context传递import uuid import apache_beam.utils.pipeline_options import apache_beam as beam runner = 'DataflowRunner' # runner = 'DirectRunner' options = beam.utils.pipeline_options.PipelineOptions() gcloud_options = options.view_as(beam.utils.pipeline_options.GoogleCloudOptions) gcloud_options.job_name = 'a' + str(uuid.uuid4()) gcloud_options.project = 'your-project' gcloud_options.staging_location = 'gs://your-bucket/beam/staging' gcloud_options.temp_location = 'gs://your-bucket/beam/temp' options.view_as(beam.utils.pipeline_options.StandardOptions).runner = runner p = beam.Pipeline(options=options) (p | 'some_strings' >> beam.Create(tuple('asdfqwert')) | 'write_text' >> beam.io.WriteToText('strings', file_name_suffix='.txt') ) p.run().wait_until_finish()。

我试图将其与最简单的代码隔离开来：

No handlers could be found for logger "oauth2client.contrib.multistore_file"
/Users/john/miniconda3/envs/py2/lib/python2.7/site-packages/apache_beam/coders/typecoders.py:136: UserWarning: Using fallback coder for typehint: Any.
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in the future. Pip now has a download command that should be used instead.
Collecting google-cloud-dataflow==0.5.1
  Using cached google-cloud-dataflow-0.5.1.tar.gz
  Saved /var/folders/v3/61xx4nnn6p36n5m9fp4qdwtr0000gn/T/tmpuCWoeh/google-cloud-dataflow-0.5.1.tar.gz
Successfully downloaded google-cloud-dataflow
Traceback (most recent call last):
  File "reproduce_bug.py", line 28, in <module>
    p.run().wait_until_finish()
  File "/Users/john/miniconda3/envs/py2/lib/python2.7/site-packages/apache_beam/runners/dataflow_runner.py", line 706, in wait_until_finish
    (self.state, getattr(self._runner, 'last_error_msg', None)), self)
apache_beam.runners.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
(70278eb56b40fd94): Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 514, in do_work
    work_executor.execute()
  File "dataflow_worker/executor.py", line 899, in dataflow_worker.executor.MapTaskExecutor.execute (dataflow_worker/executor.c:26452)
    op.start()
  File "dataflow_worker/executor.py", line 191, in dataflow_worker.executor.ReadOperation.start (dataflow_worker/executor.c:7575)
    def start(self):
  File "dataflow_worker/executor.py", line 196, in dataflow_worker.executor.ReadOperation.start (dataflow_worker/executor.c:7480)
    with self.spec.source.reader() as reader:
  File "dataflow_worker/executor.py", line 206, in dataflow_worker.executor.ReadOperation.start (dataflow_worker/executor.c:7425)
    self.output(windowed_value)
  File "dataflow_worker/executor.py", line 136, in dataflow_worker.executor.Operation.output (dataflow_worker/executor.c:5749)
    cython.cast(Receiver, self.receivers[output_index]).receive(windowed_value)
  File "dataflow_worker/executor.py", line 83, in dataflow_worker.executor.ConsumerSet.receive (dataflow_worker/executor.c:3884)
    cython.cast(Operation, consumer).process(windowed_value)
  File "dataflow_worker/executor.py", line 505, in dataflow_worker.executor.DoOperation.process (dataflow_worker/executor.c:15525)
    self.dofn_receiver.receive(o)
  File "apache_beam/runners/common.py", line 163, in apache_beam.runners.common.DoFnRunner.receive (apache_beam/runners/common.c:4862)
    self.process(windowed_value)
  File "apache_beam/runners/common.py", line 270, in apache_beam.runners.common.DoFnRunner.process (apache_beam/runners/common.c:7749)
    self.reraise_augmented(exn)
  File "apache_beam/runners/common.py", line 281, in apache_beam.runners.common.DoFnRunner.reraise_augmented (apache_beam/runners/common.c:8108)
    raise type(exn), args, sys.exc_info()[2]
  File "apache_beam/runners/common.py", line 268, in apache_beam.runners.common.DoFnRunner.process (apache_beam/runners/common.c:7660)
    self.old_dofn_process(element)
  File "apache_beam/runners/common.py", line 173, in apache_beam.runners.common.DoFnRunner.old_dofn_process (apache_beam/runners/common.c:5182)
    self._process_outputs(element, self.dofn_process(self.context))
  File "apache_beam/runners/common.py", line 152, in apache_beam.runners.common.DoFnRunner.__init__.lambda3 (apache_beam/runners/common.c:3640)
    self.dofn_process = lambda context: fn.process(context, *args)
TypeError: process() takes exactly 4 arguments (3 given) [while running 'write_text/Write/WriteImpl/WriteBundles']

完整输出：

def cubeRoot(x):
    if x >= 0:
        return x**(1/3)
    else:
        return -(-x)**(1/3)

Answer 1

您的环境似乎已安装0.5.1版本（请参阅堆栈跟踪的顶部），但您正在使用python repo的HEAD进行构建。

您可以创建具有正确版本SDK的新virtualenv环境。

如果要对Python HEAD运行，则需要在运行管道时设置sdk_location标志。
如果要针对已发布的版本运行，请使用pip install google-cloud-dataflow安装SDK，并正常运行管道。（最好在使用virtualenv的虚拟环境中）

作为一个注释，如果您使用已发布的版本，它可能是最好的。

WriteToText在DirectRunner中工作，但在DataflowRunner中失败并带有TypeError

1 个答案: