Question

我正在尝试使用为Google BigQuery提供的光束提供的I / O API，使用Apache Beam在本地（Sierra）运行管道。

我根据Beam Python quickstart的建议使用Virtualenv确定了我的环境，我可以运行wordcount.py示例。我还可以使用beam.Create和beam.ParDo正确运行自定义管道。

但我无法使用BigQuery I / O运行管道。对我做错了什么的想法？

python脚本如下。

import apache_beam as beam
from apache_beam.utils.pipeline_options import PipelineOptions
from apache_beam.io import WriteToText


class MyDoFn(beam.DoFn):
  def process(self, element):
    return element


def run():
  opts = {
    'project': 'gc-project-name'
  }
  p = beam.Pipeline(options=PipelineOptions(**opts))

  input_query = "SELECT name FROM `gc-project-name.dataset_name.table_name`"

  (p
   | beam.io.Read(beam.io.BigQuerySource(query=input_query))
   | beam.ParDo(MyDoFn())
   | beam.io.WriteToText('output.txt')
  )

  result = p.run()
  result.wait_until_finish()

if __name__ == '__main__':
  run()

当我运行它时，我收到以下错误。

WARNING:root:Task failed: Traceback (most recent call last):
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 300, in __call__
result = evaluator.finish_bundle()
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 208, in finish_bundle
with self._source.reader() as reader:
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py", line 590, in __enter__
self.client = BigQueryWrapper(client=self.test_bigquery_client)
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py", line 682, in __init__
self.client = client or bigquery.BigqueryV2(
AttributeError: 'module' object has no attribute 'BigqueryV2'
Traceback (most recent call last):
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 300, in __call__
result = evaluator.finish_bundle()
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 208, in finish_bundle
with self._source.reader() as reader:
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py", line 590, in __enter__
self.client = BigQueryWrapper(client=self.test_bigquery_client)
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py", line 682, in __init__
self.client = client or bigquery.BigqueryV2(
AttributeError: 'module' object has no attribute 'BigqueryV2'
WARNING:root:A task failed with exception.
 'module' object has no attribute 'BigqueryV2'
Traceback (most recent call last):
  File "frombigquery.py", line 54, in <module>
run()
  File "frombigquery.py", line 51, in run
result.wait_until_finish()
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py", line 157, in wait_until_finish
self._executor.await_completion()
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 335, in await_completion
self._executor.await_completion()
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 300, in __call__
result = evaluator.finish_bundle()
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 208, in finish_bundle
with self._source.reader() as reader:
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py", line 590, in __enter__
self.client = BigQueryWrapper(client=self.test_bigquery_client)
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py", line 682, in __init__
self.client = client or bigquery.BigqueryV2(
AttributeError: 'module' object has no attribute 'BigqueryV2'

Answer 1

安装Apache Beam Python SDK时，您必须添加其他选项以使用与Google Cloud Platform相关的依赖项。

pip install dist/apache-beam-*.tar.gz[gcp]

Module对象没有属性BigqueryV2 - Local Apache Beam

1 个答案: