Question

我需要在数据帧（https://pastebin.com/kNqLtUWu）中的dates上执行验证，检查date是否有效。如果date无效（即pd.to_datetime无法解析-例如0107-01-06），我需要在Fail列中填充Yes。

我对包含日期的列进行了子集处理，能够识别包含无效日期的列并将其添加到字典中，但是还没有弄清楚如何返回特定行。

我对其他方法持开放态度，但是我需要使用pandas并以Fail列结束以指示该行，我计划以此过滤最终的数据帧（一个数据帧包含日期和日期错误的行，另一个没有错误）。

有关完整代码，请参见pastebin链接

# insert empty Fail column to identify date errors
df.insert(loc=0, column='Fail', value="")

# replace all blanks with np.NaN
df.replace(r"^s*$", np.nan, regex=True, inplace = True)

# get list of date columns
cols = list(df)
date_cols = cols[2:]

# create empty dict
dfs = {}

# iterate over date columns to identify which columns contain invalid dates & add to dfs
for col in df[date_cols]:
    try:
        df[col] = df[col].apply(pd.to_datetime, errors='raise')
    except:
        print("%s column contains invalid date" % col)
        dfs[col] = df[col]

Answer 1

您所描述的问题可以通过coerce和一些逻辑来解决：

# original non_null
notnull = df[col].notnull()

# where to_datetime fails
not_datetime = pd.to_datetime(df[col], errors='coerce').isna()

not_datetime = not_datetime & notnull

Answer 2

IIUC，您所关心的是创建Fail列。因此，我专注于创建它。我认为您可以在日期时间列上使用apply，并使用自定义Lambda在axis = 1上进行切片。在将每个切片与NaN传递到pd.to_datetime之前，lambda会过滤掉coerce并从输出中检查任何NaT。

df['Fail'] = (df[date_cols].apply(lambda x: pd.to_datetime(x[x.notna()], errors='coerce')
                          .isna().any(), axis=1).replace({True: 'Fail', False: ''}))

Out[869]:
    Fail patient_ID DateOfBirth  ...    date_10    date_11     date_12
0              A001  1950-03-02  ...        NaT        NaT         NaN
1              A001  1950-03-02  ...        NaT        NaT         NaN
2              A001  1950-03-02  ...        NaT        NaT         NaN
3              A001  1950-03-02  ...        NaT        NaT         NaN
4              A001  1950-03-02  ... 2010-01-01        NaT         NaN
5              A001  1950-03-02  ...        NaT 2010-01-01         NaN
6              A001  1950-03-02  ...        NaT        NaT    1/1/2010
7              A001  1950-03-02  ...        NaT        NaT    1/1/2010
8              A001  1950-03-02  ...        NaT        NaT    1/1/2010
9              A001  1950-03-02  ...        NaT        NaT    1/1/2010
10             A001  1950-03-02  ...        NaT        NaT    1/1/2010
11             A001  1950-03-02  ...        NaT        NaT    1/1/2010
12             A001  1950-03-02  ...        NaT        NaT    1/1/2010
13             A001  1950-03-02  ...        NaT        NaT    1/1/2010
14             A001  1950-03-02  ...        NaT        NaT    1/1/2010
15  Fail       A002  1950-03-02  ...        NaT        NaT         NaN
16             A002  1950-03-02  ...        NaT        NaT         NaN
17             A002  1950-03-02  ...        NaT        NaT         NaN
18             A002  1950-03-02  ...        NaT        NaT         NaN
19             A002  1950-03-02  ... 2010-01-01        NaT         NaN
20             A002  1950-03-02  ...        NaT 2010-01-01         NaN
21             A002  1950-03-02  ...        NaT        NaT    1/1/2010
22             A002  1950-03-02  ...        NaT        NaT    1/1/2010
23             A002  1950-03-02  ...        NaT        NaT    1/1/2010
24             A002  1950-03-02  ...        NaT        NaT    1/1/2010
25             A002  1950-03-02  ...        NaT        NaT    1/1/2010
26             A002  1950-03-02  ...        NaT        NaT    1/1/2010
27             A002  1950-03-02  ...        NaT        NaT    1/1/2010
28             A002  1950-03-02  ...        NaT        NaT    1/1/2010
29  Fail       A002  1950-03-02  ...        NaT        NaT  0107-01-06

[30 rows x 15 columns]

注意：
上面的代码用于创建Fail列。不会将这些列转换为日期时间。要转换它们，您只需要分别调用pd.to_datetime。

下面是两行的值，其中Fail

In [870]: df.loc[15]
Out[870]:
Fail                          Fail
patient_ID                    A002
DateOfBirth    1950-03-02 00:00:00
date_1                  0107-01-06
date_2         2010-01-01 00:00:00
date_3                         NaT
date_4                         NaT
date_5                         NaT
date_6                         NaT
date_7                         NaT
date_8                         NaT
date_9                         NaT
date_10                        NaT
date_11                        NaT
date_12                        NaN
Name: 15, dtype: object

In [871]: df.loc[29]
Out[871]:
Fail                          Fail
patient_ID                    A002
DateOfBirth    1950-03-02 00:00:00
date_1                         NaN
date_2                         NaT
date_3                         NaT
date_4                         NaT
date_5                         NaT
date_6                         NaT
date_7                         NaT
date_8                         NaT
date_9                         NaT
date_10                        NaT
date_11                        NaT
date_12                 0107-01-06
Name: 29, dtype: object

熊猫选择发生日期时间错误的行

2 个答案: