Question

我有一个值字典，我想使用Python SDK将其写入GCS作为有效的.CSV文件。我可以将字典写成换行符分隔的文本文件，但我似乎找不到将字典转换为有效.CSV的示例。任何人都可以建议在数据流管道中生成csv的最佳方法吗？这回答了此question地址从CSV文件中读取，但并未真正解决写入CSV文件的问题。我认识到CSV文件只是带有规则的文本文件，但我仍然在努力将数据字典转换为可以使用WriteToText编写的CSV。

这是一个简单的示例字典，我想将其转换为CSV：

test_input = [{'label': 1, 'text': 'Here is a sentence'},
              {'label': 2, 'text': 'Another sentence goes here'}]


test_input  | beam.io.WriteToText(path_to_gcs)

以上将导致文本文件在换行符上包含每个字典。 Apache Beam中是否有任何我可以利用的功能（类似于csv.DictWriter）？

Answer 1

通常，您需要编写一个可以将原始dict数据元素转换为csv格式的string表示形式的函数。

该函数可以写成DoFn，您可以将其应用于您的Beam PCollection数据，这会将每个集合元素转换为所需的格式;您可以DoFn通过PCollection将ParDo应用于DoFn。您还可以将此PTransform打包为更加用户友好的# Our example list of dictionary elements test_input = [{'label': 1, 'text': 'Here is a sentence'}, {'label': 2, 'text': 'Another sentence goes here'}] def convert_my_dict_to_csv_record(input_dict): """ Turns dictionary values into a comma-separated value formatted string """ return ','.join(map(str, input_dict.values())) # Our converted list of elements converted_test_input = [convert_my_dict_to_csv_record(element) for element in test_input]。

您可以在Beam Programming Guide

中详细了解此过程

这是一个简单的，可翻译的非梁示例：

converted_test_input

['Here is a sentence,1', 'Another sentence goes here,2']如下所示：

DictWriter

使用from csv import DictWriter from csv import excel from cStringIO import StringIO ... def _dict_to_csv(element, column_order, missing_val='', discard_extras=True, dialect=excel): """ Additional properties for delimiters, escape chars, etc via an instance of csv.Dialect Note: This implementation does not support unicode """ buf = StringIO() writer = DictWriter(buf, fieldnames=column_order, restval=missing_val, extrasaction=('ignore' if discard_extras else 'raise'), dialect=dialect) writer.writerow(element) return buf.getvalue().rstrip(dialect.lineterminator) class _DictToCSVFn(DoFn): """ Converts a Dictionary to a CSV-formatted String column_order: A tuple or list specifying the name of fields to be formatted as csv, in order missing_val: The value to be written when a named field from `column_order` is not found in the input element discard_extras: (bool) Behavior when additional fields are found in the dictionary input element dialect: Delimiters, escape-characters, etc can be controlled by providing an instance of csv.Dialect """ def __init__(self, column_order, missing_val='', discard_extras=True, dialect=excel): self._column_order = column_order self._missing_val = missing_val self._discard_extras = discard_extras self._dialect = dialect def process(self, element, *args, **kwargs): result = _dict_to_csv(element, column_order=self._column_order, missing_val=self._missing_val, discard_extras=self._discard_extras, dialect=self._dialect) return [result,] class DictToCSV(PTransform): """ Transforms a PCollection of Dictionaries to a PCollection of CSV-formatted Strings column_order: A tuple or list specifying the name of fields to be formatted as csv, in order missing_val: The value to be written when a named field from `column_order` is not found in an input element discard_extras: (bool) Behavior when additional fields are found in the dictionary input element dialect: Delimiters, escape-characters, etc can be controlled by providing an instance of csv.Dialect """ def __init__(self, column_order, missing_val='', discard_extras=True, dialect=excel): self._column_order = column_order self._missing_val = missing_val self._discard_extras = discard_extras self._dialect = dialect def expand(self, pcoll): return pcoll | ParDo(_DictToCSVFn(column_order=self._column_order, missing_val=self._missing_val, discard_extras=self._discard_extras, dialect=self._dialect) )

的Beam DictToCSV DoFn和PTransform示例

test_input

要使用该示例，您可以将PCollection放入DictToCSV，并将PTransform PCollection应用于PCollection;您可以将结果转换为WriteToText并将其用作column_order的输入。请注意，您必须通过unicode参数提供与列字输入元素的键对应的列名列表或元组列;生成的CSV格式的字符串列将按提供的列名称的顺序排列。此外，该示例的基础实现不支持f :: Int -> Int f x = x + 2。

Answer 2

根据Andrew的建议，这是我创建的ConvertDictToCSV函数：

def ConvertDictToCSV(input_dict, fieldnames, separator=",", quotechar='"'):
  value_list = []
  for field in fieldnames:
    if input_dict[field]:
      field_value = str(input_dict[field])
    else:
      field_value = ""
    if separator in field_value:
      field_value = quotechar + field_value + quotechar
    value_list.append(field_value)

  return separator.join(value_list)

这似乎运作良好，但如果可能的话，使用csv.DictWriter肯定会更安全

Google Cloud Dataflow从字典

2 个答案: