Question

我有一个读取Xlsx文件的代码，对于每一行，在特定列上执行一个过程。

问题与数据流的“转换”部分有关。我实现了一个特定的方法来获取读取器发送的值，并将此数据发送到外部服务器。这个外部服务器处理数据（可能需要几分钟），然后用结果执行POST请求。（POST请求的URL在原始请求中指定。

我的问题如下：如何在外部流程完成后（外部回调）通知我的ParDo方法？

到目前为止，这是我的代码：

import logging, argparse
import apache_beam as beam
from apache_beam.io import gcsio
from apache_beam.utils.options import PipelineOptions

from openpyxl import load_workbook


# @See https://cloud.google.com/dataflow/model/custom-io-python#ptransform-wrappers
class FileReader():
    """A file reader implementation"""

    def __init__(self, path, *args, **kwargs):
        self.path = path

    def reader(self):
        return XlsxFileReader(self.path)


class XlsxFileReader():
    """The Xlsx file reader"""
    def __init__(self, path):
        self.path = path

    def _clean_value(self, value):
        if value is None:
            return None

        value = unicode(value)

        try:
            value = value.encode('utf-8')
        except UnicodeEncodeError:
            pass

        return value

    def __iter__(self):
        wb = load_workbook(filename=self.file, read_only=True)
        sheet_name = wb.get_sheet_names()[0]
        ws = wb[sheet_name]
        for line, row in enumerate(ws.rows):
            cell_value = self._clean_value(row[0].value)
            if cell_value is not None and cell_value.find('@') > 0:
                yield cell_value, line
                break

    def __enter__(self):
        self.file = gcsio.GcsIO().open(self.path, 'r')
        return self

    def __exit__(self, *args, **kwargs):
        self.file.close()


class ComputeWordLengthFn(beam.DoFn):
    def process(self, context):
        # Here, what I would need is send a request to an external API, that returns the result to the `callback` parameter.
        # I know how to do that using requests
        #
        # ***********************************************************
        # ---> BUT HOW can I know when that external service has done with my data and called back my `callback` url?
        # ***********************************************************
        yield context.element[0] is done once external service has made a request to the `callback` url on my instance.


def run(argv=None):
    parser = argparse.ArgumentParser()
    parser.add_argument(
        '--input',
        dest='input',
        default='gs://norbert-verify-staging/growthlist.xlsx',
        help='Input file to process.'
    )
    parser.add_argument(
        '--output',
        dest='output',
        required=True,
        help='Output file to write results to.'
    )
    known_args, pipeline_args = parser.parse_known_args(argv)

    pipeline_options = PipelineOptions(pipeline_args)
    p = beam.Pipeline(options=pipeline_options)

    p | 'read' >> beam.io.Read(FileReader(known_args.input)) \
      | 'verify' >> beam.ParDo(ComputeWordLengthFn()) \
      | 'write' >> beam.io.Write(beam.io.TextFileSink(known_args.output))

    p.run()


if __name__ == '__main__':
    logging.getLogger().setLevel(logging.INFO)
    run()

我希望很清楚，如果您需要更多详细信息，请与我联系。

Answer 1

不确定我是否完全理解了你的问题，但似乎你在询问Beam是否为调用给定回调后调用DoFn.process（）方法提供了一种方法。目前Beam没有提供这样的功能。

您可以在ComputeWordLengthFn.process（）方法中等待，直到完成对特定元素的请求（执行该等待的确切方式取决于外部API）。

如果我误解了你的问题，请告诉我。

Google DataFlow，如何在转换集合时等待外部webhook？

1 个答案: