Question

python -m main \ --setup_file setup.py \ --runner DataflowRunner \ --project my-test \ --staging_location gs://my-test/staging \ --temp_location gs://my-test/temp \  --template_location gs://my-test/templates/test --output gs://my-test/output

以上命令仅在本地运行（需要在本地安装依赖项），并且不会创建模板。这是main.py中的管道选项：

pipeline_options = {
    'project': 'my-test',
    'staging_location': 'gs://my-test/staging',
    'runner': 'DataflowRunner',
    'job_name': 'test',
    'temp_location': 'gs://my-test/temp',
    'save_main_session': True,
    'setup_file':'setup.py',
    'output': 'gs://my-test/output',
    'template_location': 'gs://my-test/templates/test'
}
options = PipelineOptions.from_dictionary(pipeline_options)
with beam.Pipeline(options=options) as p:

这是setup.py：

import subprocess

import setuptools
from setuptools.command.bdist_egg import bdist_egg as _bdist_egg


class bdist_egg(_bdist_egg): 
   def run(self):
      self.run_command('CustomCommands')
      _bdist_egg.run(self)
CUSTOM_COMMANDS = [
   ['apt-get', 'update'],
   ['apt-get', '--assume-yes', 'install', 'libproj-dev', 'libgdal-dev'],
   ['export' 'CPLUS_INCLUDE_PATH=/usr/include/gdal'],
   ['export' 'C_INCLUDE_PATH=/usr/include/gdal'],
   ['gdal-config', '--version'],
   ['pip', 'install', 'pygdal==1.11.3.3'],
   ['echo', 'Custom command worked!']]


class CustomCommands(setuptools.Command):


   def initialize_options(self):
      pass

   def finalize_options(self):
      pass

   def RunCustomCommand(self, command_list):

      p = subprocess.Popen(
         command_list,
         stdin=subprocess.PIPE, stdout=subprocess.PIPE, 
         stderr=subprocess.STDOUT)

      stdout_data, _ = p.communicate()

      if p.returncode != 0:
         raise RuntimeError(
          'Command %s failed: exit code: %s' % (command_list, p.returncode))

     def run(self):
        for command in CUSTOM_COMMANDS:
           self.RunCustomCommand(command)




    REQUIRED_PACKAGES = [
       'Shapely',
       'pyshp',
       'beam_utils',
       "google-cloud-storage==1.3.2",
       "google-auth",
       "requests>=2.18.0"
     ]


     setuptools.setup(
        name='ETL',
        version='0.0.1',
        description='',
        install_requires=REQUIRED_PACKAGES,
        packages=setuptools.find_packages(),
        cmdclass={

           'bdist_egg': bdist_egg,
           'CustomCommands': CustomCommands,
        }
    )

如何在具有非Python依赖项的Dataflow中创建模板？我收到的错误是

WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect.

为什么代码在本地运行而不是创建模板会在其他时间执行？请帮忙！

Answer 1

我在代码中找到了导致错误消息的位置。在管道选项中，应该是：

'setup_file':'./setup.py',

无法使用数据流+ Beam + Python创建模板

1 个答案: