Question

我有一个清单说：list=['199.72.81.55', 'burger.letters.com']。我现在想要的是从我的数据帧中获取匹配的值。例如：当我搜索burger.letters.com时，我的数据框应该返回主机，burger.letters.com的时间戳。我尝试过这样做：df.ix[host] for host in list但是，由于我有4亿行只是在df.ix[host]上执行forloop，因此需要超过30分钟。

当我在代码下运行时，它需要永远。

以下是我的数据框：

    host                     timestamp
0    199.72.81.55             01/Jul/1995:00:00:01
2    199.72.81.55             01/Jul/1995:00:00:09
3    burger.letters.com     01/Jul/1995:00:00:11
4    199.72.81.55             01/Jul/1995:00:00:12
5    199.72.81.55             01/Jul/1995:00:00:13
6    199.72.81.55             01/Jul/1995:00:00:14
8    burger.letters.com     01/Jul/1995:00:00:15
9    199.72.81.55             01/Jul/1995:00:00:15

我想要我想要的输出：

for host in hostlist:
    df.ix[host]

So this operation returns below: but too heavy as I have 0.4 billion rows. And want to optimize this.

df.ix['burger.letters.com']
       host                  timestamp
    3    burger.letters.com     01/Jul/1995:00:00:11
    8    burger.letters.com     01/Jul/1995:00:00:15

df.ix['199.72.81.55']
       host                  timestamp
    0    199.72.81.55             01/Jul/1995:00:00:01
    2    199.72.81.55             01/Jul/1995:00:00:09
    4    199.72.81.55             01/Jul/1995:00:00:12
    5    199.72.81.55             01/Jul/1995:00:00:13
    6    199.72.81.55             01/Jul/1995:00:00:14
    9    199.72.81.55             01/Jul/1995:00:00:15

以下是我的代码：//takes more than 30minutes

list(map(block, failedIP_list))

    def block(host):
        temp_df = failedIP_df.ix[host]
        if len(temp_df) > 3:
            time_values = temp_df.set_index(keys='index')['timestamp']
            if (return_seconds(time_values[2:3].values[0]) - return_seconds(time_values[0:1].values[0]))<=20:
                blocked_host.append(time_values[3:].index.tolist())

如果有人能提供帮助我真的很感激。

Answer 1

你的问题很模糊。这就是我想你想要的：

def my_function(df):
    # this function should operate on a dataframe
    # that is a subset of your original
    return dfcopy

new_df = (
    df.groupby(by=['host'])
      .filter(lambda g: g.shape[0] > 3
      .groupby(by=['host'])
      .apply(my_function)
)

groupby / filter将删除少于3个项目的组。然后我们groupby / apply对具有相同host值的所有剩余组进行操作。

熊猫：如何让我的代码运行得更快？

1 个答案: