Question

我有一个csv数据库，我正在尝试＆＃34;清理＆＃34; （使用k-anonymity替换私人信息用*＆＃39; s）。我试图选择一堆在多列中具有相同值的行。如果有足够的存在，我想改变那些行的列值。目前我有

subset= df.loc[(df['Gender'] == g) & (df['Date of birth'] == bd) & (df['Postal code'] == pc), :]
if subset.shape[0] < k :
    subset['Date of birth'] = subset['Date of birth'].apply(lambda db: f(db))

此代码产生错误

> A value is trying to be set on a copy of a slice from a DataFrame. Try
> using .loc[row_indexer,col_indexer] = value instead
> 
> See the caveats in the documentation:
> http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
> subset['Date of birth'] = subset['Date of birth'].apply(lambda db:
> day_in_date_of_birth_with_stars(db))

不确定如何解决这个问题？我可以重复查找行而不是将其存储在变量中，但这将会分配运行并希望它尽可能快。

此代码也不会修改数据框中的值

我已将代码更改为

num_with_all = df.loc[(df['Gender'] == g) & (df['Date of birth'] == bd) & (df['Postal code'] == pc)].shape[0]
if num_with_all < k:
    df.ix[(df['Gender'] == g) & (df['Date of birth'] == bd) & (df['Postal code'] == pc), 'Date of birth'] = df.loc[(df['Gender'] == g) & (df['Date of birth'] == bd) & (df['Postal code'] == pc), 'Date of birth'].apply(lambda bd: f(bd))

这似乎在运行，但对数据库中的所有子集（性别，出生日期，邮政编码）都需要很长时间。有没有办法提高效率？

示例：

我想转此

    Name    Gender  Date of birth   Telephone   Postal code Disease
0   *************   M   18-7-1981   ************    N2L 6B5 Avian Influenza
1   *********** F   28-11-1976  *********** N2L 4T6 Human Pulmonary Syndrome (HPS)
2   *************** F   4-3-1962    ************    N2L 1L9 Chlamydial infection
3   *************** F   10-8-1967   ************    N2L 4M5 Dandy fever
4   ****************    F   19-3-1963   ************    N2L 2L1 Chlamydial infection
5   ************    F   2-2-1979    ************    N2L 5J1 Scarlet fever
6   *********** M   21-1-1985   *********** N2L 1S6 Scarlet fever
7   *********** M   7-6-1977    ************    N2L 2Q9 Chlamydia
8   *************** F   9-11-1987   ************    N2L 7H9 Chlamydia
9   *****************   M   7-7-1989    ************    N2L 3B1 SARS- Severe Acute Respiratory Syndrome
10  *********** M   1-3-1969    ************    N2L 6N9 Malaria
11  **************  M   21-4-1990   ************    N2L 0B0 North American blastomycosis
12  *************** F   9-12-1964   ************    N2L 7F6 Chlamydia
13  **********  M   21-7-1960   ************    N2L 3P3 Chickenpox
14  ******************  F   11-10-1972  *********** N2L 6E4 Diphtheria
15  **************  M   25-12-1988  ************    N2L 1T4 SARS- Severe Acute Respiratory Syndrome

到

    Name    Gender  Date of birth   Telephone   Postal code Disease
0   *************   M   **-7-1981   ************    N2L 6B5 Avian Influenza
1   *********** F   **-11-1976  *********** N2L 4T6 Human Pulmonary Syndrome (HPS)
2   *************** F   *-3-1962    ************    N2L 1L9 Chlamydial infection
3   *************** F   **-8-1967   ************    N2L 4M5 Dandy fever
4   ****************    F   19-3-1963   ************    N2L 2L1 Chlamydial infection
5   ************    F   2-2-1979    ************    N2L 5J1 Scarlet fever
6   *********** M   **-1-1985   *********** N2L 1S6 Scarlet fever
7   *********** M   *-6-1977    ************    N2L 2Q9 Chlamydia
8   *************** F   9-11-1987   ************    N2L 7H9 Chlamydia
9   *****************   M   *-7-1989    ************    N2L 3B1 SARS- Severe Acute Respiratory Syndrome
10  *********** M   1-3-1969    ************    N2L 6N9 Malaria
11  **************  M   21-4-1990   ************    N2L 0B0 North American blastomycosis
12  *************** F   *-12-1964   ************    N2L 7F6 Chlamydia
13  **********  M   **-7-1960   ************    N2L 3P3 Chickenpox
14  ******************  F   11-10-1972  *********** N2L 6E4 Diphtheria
15  **************  M   **-12-1988  ************    N2L 1T4 SARS- Severe Acute Respiratory Syndrome

只有部分行的出生日期已更改，而*更换了一些数字。

Answer 1

如果提供数据，则会更容易。

执行时间清单：

df.loc [...] df.ix [...] = df.loc [...]。apply（lamda :)
循环100次 - 在0：00：02.225785中模拟1600条记录 The code in your Question

<强> LOC [...]适用。（LAMDA：系列）
循环100次 - 在0：00：00.757525

模拟1600条记录

def f2(series):
    df.loc[series.name, 'Date of birth'] = '**' + series['Date of birth'][2:]
    return series

query_df = df.loc[(df['Gender'] == g) & (df['Date of birth'] == bd) & (df['Postal code'] == pc)]

if len(query_df) < k:
    query_df.apply(lambda series: f2(series), axis=1)

loc [...]。循环索引
循环100次 - 在0：00：00.666067中模拟1600条记录

query_df = df.loc[(df['Gender'] == g) & (df['Date of birth'] == bd) & (df['Postal code'] == pc)]
if len(query_df) < k:
    for idx in query_df.index:
        df.loc[idx, 'Date of birth'] = '**' + df.loc[idx, 'Date of birth'][2:]

注意：我想知道if len(query_df) < k:，这不是if len(query_df) >= k:吗？如果您使用带有列的索引（'性别'|'出生日期'|''邮政编码'），您可以获得额外的加速。

输出 :(使用您的数据）
我只显示前5条记录并仅更改记录== 1，因为我的查询条件是 'F'，'28 -11-1976'，'N2L 4T6'

               Name Gender Date of birth     Telephone Postal code
0     *************      M     18-7-1981  ************     N2L 6B5
1       ***********      F    **-11-1976   ***********     N2L 4T6
2   ***************      F      4-3-1962  ************     N2L 1L9
3   ***************      F     10-8-1967  ************     N2L 4M5
4  ****************      F     19-3-1963  ************     N2L 2L1

使用Python测试：3.4.2 - pandas：0.19.2

Pandas Select Slice然后在列上映射函数

1 个答案: