Question

我在appache beam中编写了用于日常批处理操作的代码-数据流python代码。我正在尝试使日期范围成为现实。即目前它昨天运行良好-如果我想从开始日期运行到结束日期很困难。请您提出任何建议。请找到我昨天运行的代码段。

start_date = '20180101'
end_date = '20190101'


p = beam.Pipeline(options=options)

read = (
        p
        | 'BQRead: ' >> BQReader(
    query=test_query.format(date=date))
)

transformed = (
        read
        | 'Transform 1 ' >> beam.ParDo(Transform1())
)

transformed | 'BQWrite' >> BQWriter(table + date, table_schema)

我尝试了如下操作，但是没有用

start_date = datetime.strptime('20190101', "%Y%m%d")
end_date = datetime.strptime('20190110', "%Y%m%d")
dates = list(rrule.rrule(rrule.DAILY, dtstart=start_date, until=end_date))

for date in dates:
    ds_nd = date.strftime('%Y%m%d')

    p = beam.Pipeline(options=options)

    read = (
        p
        | 'BQRead: ' >> BQReader(
    query=test_query.format(date=ds_nd))
    )

    transformed = (
        read
        | 'Transform 1 ' >> beam.ParDo(Transform1())
    )

    transformed | 'BQWrite' >> BQWriter(table + ds_nd, table_schema)

循环数据流-用于操作的Apache Beam

0 个答案: