在两个时间序列中删除相同的异常值

Question

我有一个关于消除两次系列异常值的问题。一个时间序列包括现货市场价格，另一个包括电力输出。这两个系列是从2012年到2016年，都是CSV文件，带有时间戳，然后是值。作为功​​率输出的示例：2012-01-01 00：00：00,2335.2152646951617，价格：2012-01-01 00：00：00,17.2

由于现货市场价格非常波动并且有很多异常值，我已经过滤了它们。对于第二个时间序列，我必须删除具有相同时间戳的值，这些值在价格的时间序列中被消除。我想过生成一个包含已删除值的列表，并编写一个循环来删除第二个时间序列中具有相同时间戳的值。但到目前为止还没有奏效，我并没有真正开始。有没有人有想法？

我的python代码如下所示：

import pandas as pd
import matplotlib.pyplot as plt

power_output = pd.read_csv("./data/external/power_output.csv", delimiter=",", parse_dates=[0], index_col=[0])
print(power_output.head())
plt.plot(power_output)

spotmarket = pd.read_csv("./data/external/spotmarket_dhp.csv", delimiter=",", parse_dates=[0], index_col=[0])
print(spotmarket.head())

r = spotmarket['price'].pct_change().dropna() * 100
print(r)
plt.plot(r)

Q1 = r.quantile(.25)
Q3 = r.quantile(.75)
q1 = Q1-2*(Q3-Q1)
q3 = Q3+2*(Q3-Q1)

a = r[r.between(q1, q3)]
print(a)
plt.plot(a)

有人能帮助我吗？

Answer 1

如果您的问题是如何比较两个时间戳，可以查看this。

基本上你可以这样做：

out = r[~r.between(q1, q3)] # negation of your between to get the outliers
df=pd.merge(spotmarker,out,on=['date'],how="outer",indicator=True)
df=df[df['_merge']=='left_only']

这是一种合并操作，只保留那些仅存在于左数据框中的行

Answer 2

以下建议基于previous post的答案。 您可以通过merging两个系列解决问题并将其存储在pandas dataframe中。然后，您可以使用任何所需的技术来识别和删除异常值。看看上面提到的帖子。

以下是使用可处理多个系列的代码段来解决您的特定问题：

由于我无法访问您的数据，因此以下代码段将生成两个系列，其中一个系列具有独特的异常值：

def sample(colname):

    base = 100
    nsample = 20
    sigma = 10

    # Basic df with trend and sinus seasonality 
    trend1 = np.linspace(0,1, nsample)
    y1 = np.sin(trend1)
    dates = pd.date_range(pd.datetime(2016, 1, 1).strftime('%Y-%m-%d'), periods=nsample).tolist()
    df = pd.DataFrame({'dates':dates, 'trend1':trend1, 'y1':y1})
    df = df.set_index(['dates'])
    df.index = pd.to_datetime(df.index)

    # Gaussian Noise with amplitude sigma
    df['y2'] = sigma * np.random.normal(size=nsample)
    df['y3'] = df['y2'] + base + (np.sin(trend1))
    df['trend2'] = 1/(np.cos(trend1)/1.05)
    df['y4'] = df['y3'] * df['trend2']

    df=df['y4'].to_frame()
    df.columns = [colname]

    return(df)

df_sample1 = sample(colname = 'series1')
df_sample2 = sample(colname = 'series2')
df_sample2['series2'].iloc[10] = 800
df_sample1.plot()
df_sample2.plot()

系列1 - 没有异常值

系列2 - 一个独特的异常值

现在你可以像这样合并这些系列：

# Merge dataframes
df_merged = pd.merge(df_sample1, df_sample2, how='outer', left_index=True, right_index=True)
df_merged.plot()

什么被视为异常值将完全取决于数据集的性质。在这种情况下，您可以使用sscipy.zscore()设置识别异常值的级别。在下列情况下，每个差异超过3的观察都被视为异常值。

# A function for removing outliers
def noSpikes(df, level, keepFirst):

    # 1. Get some info about the original data:

    ##%%
    #df = df_merged
    #level = 3
    #keepFirst = True
    ##%%

    firstVal = df[:1]
    colNames = df.columns
    colNumber = len(df.columns)

    #cleanBy = 'Series1'

    # 2. Take the first difference and 
    df_diff = df.diff()

    # 3. Remove missing values
    df_clean = df_diff.dropna()

    # 4. Select a level for a Z-score to identify and remove outliers
    df_Z = df_clean[(np.abs(stats.zscore(df_clean)) < level).all(axis=1)]
    ix_keep = df_Z.index

    # 5. Subset the raw dataframe with the indexes you'd like to keep
    df_keep = df.loc[ix_keep]

    # 6. 
    # df_keep will be missing some indexes.
    # Do the following if you'd like to keep those indexes
    # and, for example, fill missing values with the previous values
    df_out = pd.merge(df_keep, df, how='outer', left_index=True, right_index=True)

    # 7. Keep only the original columns (drop the diffs)
    df_out = df_out.ix[:,:colNumber]

    # 8. Fill missing values
    df_complete = df_out.fillna(axis=0, method='ffill')

    # 9. Reset column names
    df_complete.columns = colNames

    # Keep the first value
    if keepFirst:
        df_complete.iloc[0] = firstVal.iloc[0]

    return(df_complete)

df_clean = noSpikes(df = df_merged, level = 3, keepFirst = True)
df_clean.plot()

让我知道这对你有用。

这里有一个简单的复制粘贴的全部内容：

# Imports
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
from scipy import stats

np.random.seed(22)

# A function for noisy data with a trend element
def sample(colname):

    base = 100
    nsample = 20
    sigma = 10

    # Basic df with trend and sinus seasonality 
    trend1 = np.linspace(0,1, nsample)
    y1 = np.sin(trend1)
    dates = pd.date_range(pd.datetime(2016, 1, 1).strftime('%Y-%m-%d'), periods=nsample).tolist()
    df = pd.DataFrame({'dates':dates, 'trend1':trend1, 'y1':y1})
    df = df.set_index(['dates'])
    df.index = pd.to_datetime(df.index)

    # Gaussian Noise with amplitude sigma
    df['y2'] = sigma * np.random.normal(size=nsample)
    df['y3'] = df['y2'] + base + (np.sin(trend1))
    df['trend2'] = 1/(np.cos(trend1)/1.05)
    df['y4'] = df['y3'] * df['trend2']

    df=df['y4'].to_frame()
    df.columns = [colname]

    return(df)

df_sample1 = sample(colname = 'series1')
df_sample2 = sample(colname = 'series2')
df_sample2['series2'].iloc[10] = 800
df_sample1.plot()
df_sample2.plot()

# Merge dataframes
df_merged = pd.merge(df_sample1, df_sample2, how='outer', left_index=True, right_index=True)
df_merged.plot()

# A function for removing outliers
def noSpikes(df, level, keepFirst):

    # 1. Get some info about the original data:
    firstVal = df[:1]
    colNames = df.columns
    colNumber = len(df.columns)

    #cleanBy = 'Series1'

    # 2. Take the first difference and 
    df_diff = df.diff()

    # 3. Remove missing values
    df_clean = df_diff.dropna()

    # 4. Select a level for a Z-score to identify and remove outliers
    df_Z = df_clean[(np.abs(stats.zscore(df_clean)) < level).all(axis=1)]
    ix_keep = df_Z.index

    # 5. Subset the raw dataframe with the indexes you'd like to keep
    df_keep = df.loc[ix_keep]

    # 6. 
    # df_keep will be missing some indexes.
    # Do the following if you'd like to keep those indexes
    # and, for example, fill missing values with the previous values
    df_out = pd.merge(df_keep, df, how='outer', left_index=True, right_index=True)

    # 7. Keep only the original columns (drop the diffs)
    df_out = df_out.ix[:,:colNumber]

    # 8. Fill missing values
    df_complete = df_out.fillna(axis=0, method='ffill')

    # 9. Reset column names
    df_complete.columns = colNames

    # Keep the first value
    if keepFirst:
        df_complete.iloc[0] = firstVal.iloc[0]

    return(df_complete)

df_clean = noSpikes(df = df_merged, level = 3, keepFirst = True)
df_clean.plot()