Question

我有一个使用db.SelfReferenceProperty创建树状结构的类。

尝试使用appcfg.py upload_data -- config_file=bulkloader.yaml --kind=Group --filename=group.csv (...)填充数据库时，我有一个例外BadValueError: name must not be empty。 ^{（下面的完整堆栈）}

我尝试对数据进行排序，以确保具有指向它们的外键的组是第一个。那没用。

通过在bulkloader.yaml中注释进行转换的行“import_transform：transform.create_foreign_key（'Group'）”，数据被上传，但它将该属性存储为字符串，从而破坏了我的应用程序逻辑。

- kind: Group
  connector: csv
  connector_options:
  property_map:
    - property: __key__
      external_name: key
      export_transform: transform.key_id_or_name_as_string

    - property: name
      external_name: name
      # Type: String Stats: 9 properties of this type in this kind.

    - property: section
      external_name: section
      # Type: Key Stats: 6 properties of this type in this kind.
      import_transform: transform.create_foreign_key('Group')
      export_transform: transform.key_id_or_name_as_string

有没有办法让批量加载程序考虑自引用，或者我应该在批量加载数据的服务器端转换还是实现我自己的批量加载算法？

----
Traceback (most recent call last):
  File "/home/username/src/google_appengine/google/appengine/tools/adaptive_thread_pool.py", line 150, in WorkOnItems
    status, instruction = item.PerformWork(self.__thread_pool)
  File "/home/username/src/google_appengine/google/appengine/tools/bulkloader.py", line 691, in PerformWork
    transfer_time = self._TransferItem(thread_pool)
  File "/home/username/src/google_appengine/google/appengine/tools/bulkloader.py", line 846, in _TransferItem
    self.content = self.request_manager.EncodeContent(self.rows)
  File "/home/username/src/google_appengine/google/appengine/tools/bulkloader.py", line 1267, in EncodeContent
    entity = loader.create_entity(values, key_name=key, parent=parent)
  File "/home/username/src/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 382, in create_entity
    return self.dict_to_entity(input_dict, self.bulkload_state)
  File "/home/username/src/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 133, in dict_to_entity
    self.__run_import_transforms(input_dict, instance, bulkload_state_copy)
  File "/home/username/src/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 230, in __run_import_transforms
    value = self.__dict_to_prop(transform, input_dict, bulkload_state)
  File "/home/username/src/google_appengine/google/appengine/ext/bulkload/bulkloader_config.py", line 188, in __dict_to_prop
    value = transform.import_transform(value)
  File "/home/username/src/google_appengine/google/appengine/ext/bulkload/bulkloader_parser.py", line 93, in __call__
    return self.method(*args, **kwargs)
  File "/home/username/src/google_appengine/google/appengine/ext/bulkload/transform.py", line 114, in generate_foreign_key_lambda
        return datastore.Key.from_path(kind, value)
  File "/home/username/src/google_appengine/google/appengine/api/datastore_types.py", line 384, in from_path
    ValidateString(id_or_name, 'name')
  File "/home/username/src/google_appengine/google/appengine/api/datastore_types.py", line 109, in ValidateString
    raise exception('%s must not be empty.' % name)
BadValueError: name must not be empty.

Answer 1

使用an answer given to a similar problem作为基础，我可以通过创建一个小helpers.py文件作为transform.create_foreign_key的包装来成功解决此问题：

from google.appengine.api import datastore

def create_foreign_key(kind, key_is_id=False):
  def generate_foreign_key_lambda(value):
    if value is None:
      return None

    if key_is_id:
      value = int(value)
    try:
      return datastore.Key.from_path(kind, value)
    except:
      return None

  return generate_foreign_key_lambda

一旦该文件与yaml bolk上传配置文件（bulkloader.yaml）位于同一目录中，就将其添加到该文件中：

python_preamble:
- (...)
- import: helpers

transformers:

- kind: Group
  connector: csv
  connector_options:
  property_map:
    - property: __key__
      external_name: key
      export_transform: transform.key_id_or_name_as_string

    - property: name
      external_name: name

    - property: section
      external_name: section
      import_transform: helpers.create_foreign_key('Group')
                      # ^^^^^^^ we use the wrapper instead
      export_transform: transform.key_id_or_name_as_string

通过这些更改，批量上传现在可以正常运行。

在使用之前，您一定要修改全部except，并将其替换为except BadValueError。

Answer 2

transform.py（也许只是最近）包含解决此问题的装饰器：

def none_if_empty(fn):
  """A decorator which returns None if its input is empty else fn(x).

  Useful on import.  Can be used in config files
  (e.g. "transform.none_if_empty(int)" or as a decorator.

  Args:
    fn: Single argument transform function.

  Returns:
    Wrapped function.
  """

  def wrapper(value):


    if value == '' or value is None or value == []:
      return None
    return fn(value)

  return wrapper

因此，如果不引入自定义helpers.py文件，使用以下内容也可以解决问题：

transform.none_if_empty(transform.create_foreign_key('Group'))

如何使用bulkuploader使用db.SelfReferenceProperty填充类？

2 个答案: