如何在dask.dataframe中的多列中爆炸dict(或dict列表)对象

时间:2019-08-29 13:57:33

标签: python pandas dask xmltodict

当我尝试使用xmltodict将一些xml转换为dataframe时,碰巧一个特定的列包含了我需要作为dict或dict列表的所有信息。我可以使用pandas将该列转换成多个列,但不能在dask中执行类似的操作。

不可能使用meta,因为我不知道xml中所有可用的字段,并且dask是必需的,因为真正的xml文件每个都大于1Gb。

example.xml:

<?xml version="1.0" encoding="UTF-8"?>
<itemList>
  <eventItem uid="1">
    <timestamp>2019-07-04T09:57:35.044Z</timestamp>
    <eventType>generic</eventType>
    <details>
      <detail>
        <name>columnA</name>
        <value>AAA</value>
      </detail>
      <detail>
        <name>columnB</name>
        <value>BBB</value>
      </detail>
    </details>
  </eventItem>
  <eventItem uid="2">
    <timestamp>2019-07-04T09:57:52.188Z</timestamp>
    <eventType>generic</eventType>
    <details>
      <detail>
        <name>columnC</name>
        <value>CCC</value>
      </detail>
    </details>
  </eventItem>
</itemList>

工作熊猫代码:

import xmltodict
import collections
import pandas as pd

def pd_output_dict(details):
    detail = details.get("detail", [])
    ret_value = {}
    if type(detail) in (collections.OrderedDict, dict):
        ret_value[detail["name"]] = detail["value"]
    elif type(detail) == list:
        for i in detail:
            ret_value[i["name"]] = i["value"]
    return pd.Series(ret_value)

with open("example.xml", "r", encoding="utf8") as f:
    df_dict_list = xmltodict.parse(f.read()).get("itemList", {}).get("eventItem", [])
    df = pd.DataFrame(df_dict_list)
    df = pd.concat([df, df.apply(lambda row: pd_output_dict(row.details), axis=1, result_type="expand")], axis=1)
    print(df.head())

无效的代码:

import xmltodict
import collections
import dask
import dask.bag as db
import dask.dataframe as dd

def dd_output_dict(row):
    detail = row.get("details", {}).get("detail", [])
    ret_value = {}
    if type(detail) in (collections.OrderedDict, dict):
        row[detail["name"]] = detail["value"]
    elif type(detail) == list:
        for i in detail:
            row[i["name"]] = i["value"]
    return row

with open("example.xml", "r", encoding="utf8") as f:
    df_dict_list = xmltodict.parse(f.read()).get("itemList", {}).get("eventItem", [])
    df_bag = db.from_sequence(df_dict_list)
    df = df_bag.to_dataframe()
    df = df.apply(lambda row: dd_output_dict(row), axis=1)

这个想法是让我在大熊猫中有类似的结果,但是在我收到错误的那一刻:

>>> df = df.apply(lambda row: output_dict(row), axis=1)
Traceback (most recent call last):
  File "C:\Anaconda3\lib\site-packages\dask\dataframe\utils.py", line 169, in raise_on_meta_error
    yield
  File "C:\Anaconda3\lib\site-packages\dask\dataframe\core.py", line 4711, in _emulate
    return func(*_extract_meta(args, True), **_extract_meta(kwargs, True))
  File "C:\Anaconda3\lib\site-packages\dask\utils.py", line 854, in __call__
    return getattr(obj, self.method)(*args, **kwargs)
  File "C:\Anaconda3\lib\site-packages\pandas\core\frame.py", line 6487, in apply
    return op.get_result()
  File "C:\Anaconda3\lib\site-packages\pandas\core\apply.py", line 151, in get_result
    return self.apply_standard()
  File "C:\Anaconda3\lib\site-packages\pandas\core\apply.py", line 257, in apply_standard
    self.apply_series_generator()
  File "C:\Anaconda3\lib\site-packages\pandas\core\apply.py", line 286, in apply_series_generator
    results[i] = self.f(v)
  File "<stdin>", line 1, in <lambda>
  File "<stdin>", line 4, in output_dict
AttributeError: ("'str' object has no attribute 'get'", 'occurred at index 0')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Anaconda3\lib\site-packages\dask\dataframe\core.py", line 3964, in apply
    M.apply, self._meta_nonempty, func, args=args, udf=True, **kwds
  File "C:\Anaconda3\lib\site-packages\dask\dataframe\core.py", line 4711, in _emulate
    return func(*_extract_meta(args, True), **_extract_meta(kwargs, True))
  File "C:\Anaconda3\lib\contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Anaconda3\lib\site-packages\dask\dataframe\utils.py", line 190, in raise_on_meta_error
    raise ValueError(msg)
ValueError: Metadata inference failed in `apply`.

You have supplied a custom function and Dask is unable to
determine the type of output that that function returns.

To resolve this please provide a meta= keyword.
The docstring of the Dask function you ran should have more information.

Original error is below:
------------------------
AttributeError("'str' object has no attribute 'get'", 'occurred at index 0')

Traceback:
---------
  File "C:\Anaconda3\lib\site-packages\dask\dataframe\utils.py", line 169, in raise_on_meta_error
    yield
  File "C:\Anaconda3\lib\site-packages\dask\dataframe\core.py", line 4711, in _emulate
    return func(*_extract_meta(args, True), **_extract_meta(kwargs, True))
  File "C:\Anaconda3\lib\site-packages\dask\utils.py", line 854, in __call__
    return getattr(obj, self.method)(*args, **kwargs)
  File "C:\Anaconda3\lib\site-packages\pandas\core\frame.py", line 6487, in apply
    return op.get_result()
  File "C:\Anaconda3\lib\site-packages\pandas\core\apply.py", line 151, in get_result
    return self.apply_standard()
  File "C:\Anaconda3\lib\site-packages\pandas\core\apply.py", line 257, in apply_standard
    self.apply_series_generator()
  File "C:\Anaconda3\lib\site-packages\pandas\core\apply.py", line 286, in apply_series_generator
    results[i] = self.f(v)
  File "<stdin>", line 1, in <lambda>
  File "<stdin>", line 4, in output_dict

1 个答案:

答案 0 :(得分:0)

对,因此诸如map_partitions之类的操作将需要知道列名和数据类型。如前所述,您可以使用meta=关键字进行指定。

也许您可以遍历数据一次以计算出它们是什么,然后构造一个适当的元对象并将其传递给?这效率低下,需要通读所有数据,但是我不确定是否还有另一种方法。