Question

我正在尝试从包含连续数据区域的数据帧中创建张量。

我进行的尝试利用了类似于df.groupby(groupcol)[aggcol].agg(list)类型的命令的groupby / aggregate，它基于对单独的列（groupcol）进行分组而轻松获取列（aggcol）的列表

以下是两个输入数据帧（特征及其元数据）以及要处理的结果数据帧（由元数据和位置标注的特征）的示例：

我正在将这些功能分组到每个区域的列表中：

但这需要为每个区域填充一定的长度（例如，通过在带注释的要素数据框中为每个区域级别添加行）。

换句话说，按区域分组后的结果数据帧将是：

这样我就可以以这种格式返回此数据框中的值的数组：

array([[[2.965e+03, 4.800e-01],
    [4.894e+03, 8.700e-01],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00]],

   [[7.920e+02, 1.700e-01],
    [3.029e+03, 8.100e-01],
    [4.852e+03, 7.400e-01],
    [9.548e+03, 6.000e-01],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00]],

   [[2.469e+03, 3.600e-01],
    [7.144e+03, 1.600e-01],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00]],

   [[5.783e+03, 7.000e-01],
    [7.068e+03, 6.000e-01],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00]],

   [[2.965e+03, 9.800e-01],
    [4.894e+03, 8.900e-01],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00]],

   [[7.920e+02, 8.600e-01],
    [3.029e+03, 8.600e-01],
    [4.852e+03, 6.900e-01],
    [9.548e+03, 5.900e-01],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00]],

   [[2.469e+03, 6.700e-01],
    [7.144e+03, 1.300e-01],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00]],

   [[5.783e+03, 8.400e-01],
    [7.068e+03, 9.900e-01],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00]],

   [[2.965e+03, 8.000e-02],
    [4.894e+03, 5.700e-01],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00]],

   [[7.920e+02, 4.000e-01],
    [3.029e+03, 1.100e-01],
    [4.852e+03, 8.000e-01],
    [9.548e+03, 3.400e-01],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00]],

   [[2.469e+03, 1.800e-01],
    [7.144e+03, 6.300e-01],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00]],

   [[5.783e+03, 4.700e-01],
    [7.068e+03, 3.200e-01],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00],
    [0.000e+00, 0.000e+00]]])

以下是用于生成输入表的代码：

import numpy as np
import pandas as pd

# DataFrame of Features in each document
num_features = 10
num_docs = 3
def calc_feats(): 
    return np.random.randint(0, 100, num_features)/100
d = {'document_'+str(i): calc_feats() for i in range(0, num_docs)}
# Unique feature index
d['feat_index'] = np.arange(0, num_features)
docs = pd.DataFrame(d)\
                .set_index('feat_index')

# DataFrame for metadata about features
regions_of_doc = ['mid', 'end', 'start', 'intro', 'title']
feature_regions = [np.random.choice(regions_of_doc) for i in range(0, num_features)]
feature_positions = np.random.randint(0, 10000, num_features)
feature_meta_info = pd.DataFrame({'feat_index': d['feat_index'],
                                'region': feature_regions,
                                'position_in_region': feature_positions,
                                'other_uninteresting_info': np.random.randint(0, 10000, num_times)})\
                    .set_index('feat_index')


# Join the two dataframes and set a multi-index to annotate the documents
combined_df = docs.join(feature_meta_info.drop('other_uninteresting_info', axis = 1))\
                    .reset_index()\
                    .set_index(['region', 'feat_index', 'position_in_region'])\
                    .sort_index(level = ['region', 'position_in_region'])

# add position features to each feature
pos = combined_df.index.get_level_values('position_in_region')
combined_df = combined_df.apply(lambda x: list(zip(pos, x)))

# display(multi_table([docs, feature_meta_info, ]))
display(HTML('<table><tr style="background-color:white;">' + \
             '<td>' + docs._repr_html_() + '</td>' + \
             '<td><img src = "https://upload.wikimedia.org/wikipedia/commons/9/9e/Plus_symbol.svg", width = "50", height = "50"></td>' + \
             '<td>' + feature_meta_info._repr_html_() + '</td>' + \
             '<td><img src = "https://upload.wikimedia.org/wikipedia/commons/thumb/7/71/Arrow_east.svg/800px-Arrow_east.svg.png", width = "50", height = "10"></td>' + \
             '<td>' + combined_df._repr_html_() + '</td>' + \
             '</tr></table>'))

Answer 1

在编写了一些最小，完整和可验证的示例和类比之后，我有了一些有效的代码。它可能没有性能（即将检查），但这是使用大量stack() sort_index()和groupby()函数的一种方式：

# stack documents into series 
# and then and order the index by document first, then region and position
featvals = combined_df.stack()
featvals.index.set_names(['region','feat_index','position_in_region', 'document'], inplace = True)
featvals = featvals.reorder_levels(order = ['document', 'region', 'position_in_region', 'feat_index'])\
                       .sort_index(level = ['document', 'region', 'position_in_region'])

display(featvals.to_frame())

# Group into lists by each document and region for each feature tuple
feat_by_region = featvals.groupby(level = ['document', 'region']).agg(list)

display(feat_by_region.to_frame())

def pad_lists(list_of_arrays, max_seq_len, null_value):
    arr = np.array([list(i) + [null_value] * (max_seq_len - len(i)) for i in list_of_arrays])
    return  arr

# Solution:
# Numpy array by getting the lists by `.values` and pad these arrays to a standardized length
feat_by_region_array = pad_lists(feat_by_region.values, max_seq_len = 7, null_value = (0,0))



# dataframe to view the feature array
feat_by_region = pd.DataFrame([tuple(i) for i in feat_by_region_array[..., :]], index = feat_by_region.index)
feat_by_region.columns.name = 'position_index'
display(feat_by_region)

# doesn't condense to properly formatted array (- because of tuples?)
display(np.array(feat_by_region.values).shape)

输出：

堆叠/订购：

分组到列表：

填充后的值的数据框：

大熊猫聚合和分组：将多列分组，排序，填充和存储数组

1 个答案: