Question

我在使用dask时遇到了严重问题（dask版本：1.00，pandas版本：0.23.3）。我正在尝试从CSV文件加载dask数据帧，将结果过滤到两个单独的数据帧中，然后对两者进行操作。

但是，在拆分数据框并尝试将类别列设置为“已知”后，它们仍为“未知”。因此，我无法继续进行操作（这要求类别列为“已知”。）

注意：：我已经建议使用熊猫创建一个最小示例，而不是使用read_csv（）。

import pandas as pd
import dask.dataframe as dd

# Specify dtypes
b_dtypes = {
    'symbol': 'category',
    'price': 'float64',
}

i_dtypes = {
    'symbol': 'category',
    'price': 'object'
}

# Specify a function to quickly set dtypes
def to_dtypes(df, dtypes):
    for column, dtype in dtypes.items():
        if column in df.columns:
            df[column] = df.loc[:, column].astype(dtype)
    return df

# Set up our test data
data = [
    ['B', 'IBN', '9.9800'],
    ['B', 'PAY', '21.5000'],
    ['I', 'PAY', 'seventeen'],
    ['I', 'SPY', 'ten']
]

# Create pandas dataframe
pdf = pd.DataFrame(data, columns=['type', 'symbol', 'price'], dtype='object')

# Convert into dask
df = dd.from_pandas(pdf, npartitions=3)

#
## At this point 'df' simulates what I get when I read the mixed-type CSV file via dask
#

# Split the dataframe by the 'type' column
b_df = df.loc[df['type'] == 'B', :]
i_df = df.loc[df['type'] == 'I', :]

# Convert columns into our intended dtypes
b_df = to_dtypes(b_df, b_dtypes)
i_df = to_dtypes(i_df, i_dtypes)

# Let's convert our 'symbol' column to known categories
b_df = b_df.categorize(columns=['symbol'])
i_df['symbol'] = i_df['symbol'].cat.as_known()

# Is our symbol column known now?
print(b_df['symbol'].cat.known, flush=True)
print(i_df['symbol'].cat.known, flush=True)

#
## print() returns 'False' for both, this makes me want to kill myself.
## (Please help...)
#

更新：看来，如果我将'npartitions'参数移到1，则在两种情况下print（）都会返回True。因此，对于包含不同类别的分区，这似乎是一个问题。但是，将两个数据帧仅加载到两个分区中是不可行的，那么有没有办法让dask进行某种重新排序以使各个分区中的类别保持一致？

Answer 1

您的问题的答案基本上包含在doc中。我指的是# categorize requires computation, and results in known categoricals注释的零件代码，因为我似乎误用了loc

import pandas as pd
import dask.dataframe as dd

# Set up our test data
data = [['B', 'IBN', '9.9800'],
        ['B', 'PAY', '21.5000'],
        ['I', 'PAY', 'seventeen'],
        ['I', 'SPY', 'ten']
       ]

# Create pandas dataframe
pdf = pd.DataFrame(data, columns=['type', 'symbol', 'price'], dtype='object')

# Convert into dask
ddf = dd.from_pandas(pdf, npartitions=3)

# Split the dataframe by the 'type' column
# reset_index is not necessary
b_df = ddf[ddf["type"] == "B"].reset_index(drop=True)
i_df = ddf[ddf["type"] == "I"].reset_index(drop=True)

# Convert columns into our intended dtypes
b_df = b_df.categorize(columns=['symbol'])
b_df["price"] = b_df["price"].astype('float64')
i_df = i_df.categorize(columns=['symbol'])

# Is our symbol column known now? YES
print(b_df['symbol'].cat.known, flush=True)
print(i_df['symbol'].cat.known, flush=True)

使用.loc后，Dask categorize（）将不起作用

1 个答案: