如何有效地计算熊猫时间序列中的滚动唯一计数?

时间:2017-09-28 13:44:13

标签: python pandas time-series distinct-values rolling-computation

我有一系列时间访问建筑物的人。每个人都有一个唯一的身份证。对于时间序列中的每条记录,我想知道在过去365天内访问建筑物的唯一人数(即滚动的唯一计数,窗口为365天)。

pandas似乎没有内置的计算方法。当存在大量唯一访问者和/或大窗口时,计算变得计算密集。 (实际数据大于此示例。)

有没有比我在下面做的更好的计算方法?我不确定为什么我所做的快速方法,windowed_nunique(在“速度测试3”下)是1。

感谢您的帮助!

相关链接:

初始化

In [1]:

# Import libraries.
import pandas as pd
import numba
import numpy as np

In [2]:

# Create data of people visiting a building.

np.random.seed(seed=0)
dates = pd.date_range(start='2010-01-01', end='2015-01-01', freq='D')
window = 365 # days
num_pids = 100
probs = np.linspace(start=0.001, stop=0.1, num=num_pids)

df = pd\
    .DataFrame(
        data=[(date, pid)
              for (pid, prob) in zip(range(num_pids), probs)
              for date in np.compress(np.random.binomial(n=1, p=prob, size=len(dates)), dates)],
        columns=['Date', 'PersonId'])\
    .sort_values(by='Date')\
    .reset_index(drop=True)

print("Created data of people visiting a building:")
df.head() # 9181 rows × 2 columns

Out[2]:

Created data of people visiting a building:

|   | Date       | PersonId | 
|---|------------|----------| 
| 0 | 2010-01-01 | 76       | 
| 1 | 2010-01-01 | 63       | 
| 2 | 2010-01-01 | 89       | 
| 3 | 2010-01-01 | 81       | 
| 4 | 2010-01-01 | 7        | 

速度参考

In [3]:

%%timeit
# This counts the number of people visiting the building, not the number of unique people.
# Provided as a speed reference.
df.rolling(window='{:d}D'.format(window), on='Date').count()

3.32 ms ± 124 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

速度测试1

In [4]:

%%timeit
df.rolling(window='{:d}D'.format(window), on='Date').apply(lambda arr: pd.Series(arr).nunique())

2.42 s ± 282 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [5]:

# Save results as a reference to check calculation accuracy.
ref = df.rolling(window='{:d}D'.format(window), on='Date').apply(lambda arr: pd.Series(arr).nunique())['PersonId'].values

速度测试2

In [6]:

# Define a custom function and implement a just-in-time compiler.
@numba.jit(nopython=True)
def nunique(arr):
    return len(set(arr))

In [7]:

%%timeit
df.rolling(window='{:d}D'.format(window), on='Date').apply(nunique)

430 ms ± 31.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [8]:

# Check accuracy of results.
test = df.rolling(window='{:d}D'.format(window), on='Date').apply(nunique)['PersonId'].values
assert all(ref == test)

速度测试3

In [9]:

# Define a custom function and implement a just-in-time compiler.
@numba.jit(nopython=True)
def windowed_nunique(dates, pids, window):
    r"""Track number of unique persons in window,
    reading through arrays only once.

    Args:
        dates (numpy.ndarray): Array of dates as number of days since epoch.
        pids (numpy.ndarray): Array of integer person identifiers.
        window (int): Width of window in units of difference of `dates`.

    Returns:
        ucts (numpy.ndarray): Array of unique counts.

    Raises:
        AssertionError: Raised if `len(dates) != len(pids)`

    Notes:
        * May be off by 1 compared to `pandas.core.window.Rolling`
            with a time series alias offset.

    """

    # Check arguments.
    assert dates.shape == pids.shape

    # Initialize counters.
    idx_min = 0
    idx_max = dates.shape[0]
    date_min = dates[idx_min]
    pid_min = pids[idx_min]
    pid_max = np.max(pids)
    pid_cts = np.zeros(pid_max, dtype=np.int64)
    pid_cts[pid_min] = 1
    uct = 1
    ucts = np.zeros(idx_max, dtype=np.int64)
    ucts[idx_min] = uct
    idx = 1

    # For each (date, person)...
    while idx < idx_max:

        # If person count went from 0 to 1, increment unique person count.
        date = dates[idx]
        pid = pids[idx]
        pid_cts[pid] += 1
        if pid_cts[pid] == 1:
            uct += 1

        # For past dates outside of window...
        while (date - date_min) > window:

            # If person count went from 1 to 0, decrement unique person count.
            pid_cts[pid_min] -= 1
            if pid_cts[pid_min] == 0:
                uct -= 1
            idx_min += 1
            date_min = dates[idx_min]
            pid_min = pids[idx_min]

        # Record unique person count.
        ucts[idx] = uct
        idx += 1

    return ucts

In [10]:

# Cast dates to integers.
df['DateEpoch'] = (df['Date'] - pd.to_datetime('1970-01-01'))/pd.to_timedelta(1, unit='D')
df['DateEpoch'] = df['DateEpoch'].astype(int)

In [11]:

%%timeit
windowed_nunique(
    dates=df['DateEpoch'].values,
    pids=df['PersonId'].values,
    window=window)

107 µs ± 63.5 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [12]:

# Check accuracy of results.
test = windowed_nunique(
    dates=df['DateEpoch'].values,
    pids=df['PersonId'].values,
    window=window)
# Note: Method may be off by 1.
assert all(np.isclose(ref, np.asarray(test), atol=1))

In [13]:

# Show where the calculation doesn't match.
print("Where reference ('ref') calculation of number of unique people doesn't match 'test':")
df['ref'] = ref
df['test'] = test
df.loc[df['ref'] != df['test']].head() # 9044 rows × 5 columns

Out[13]:

Where reference ('ref') calculation of number of unique people doesn't match 'test':

|    | Date       | PersonId | DateEpoch | ref  | test | 
|----|------------|----------|-----------|------|------| 
| 78 | 2010-01-19 | 99       | 14628     | 56.0 | 55   | 
| 79 | 2010-01-19 | 96       | 14628     | 56.0 | 55   | 
| 80 | 2010-01-19 | 88       | 14628     | 56.0 | 55   | 
| 81 | 2010-01-20 | 94       | 14629     | 56.0 | 55   | 
| 82 | 2010-01-20 | 48       | 14629     | 57.0 | 56   | 

3 个答案:

答案 0 :(得分:1)

我在快速方法windowed_nunique中遇到了2个错误,现在已在以下windowed_nunique_corrected中更正:

  1. 用于记忆窗口pid_cts内每个人ID的唯一计数数量的数组的大小太小。
  2. 由于窗口的前端和后端包含整数天,因此date_min时应更新(date - date_min + 1) > window
  3. 相关链接:

    In [14]:

    # Define a custom function and implement a just-in-time compiler.
    @numba.jit(nopython=True)
    def windowed_nunique_corrected(dates, pids, window):
        r"""Track number of unique persons in window,
        reading through arrays only once.
    
        Args:
            dates (numpy.ndarray): Array of dates as number of days since epoch.
            pids (numpy.ndarray): Array of integer person identifiers.
                Required: min(pids) >= 0
            window (int): Width of window in units of difference of `dates`.
                Required: window >= 1
    
        Returns:
            ucts (numpy.ndarray): Array of unique counts.
    
        Raises:
            AssertionError: Raised if not...
                * len(dates) == len(pids)
                * min(pids) >= 0
                * window >= 1
    
        Notes:
            * Matches `pandas.core.window.Rolling`
                with a time series alias offset.
    
        """
    
        # Check arguments.
        assert len(dates) == len(pids)
        assert np.min(pids) >= 0
        assert window >= 1
    
        # Initialize counters.
        idx_min = 0
        idx_max = dates.shape[0]
        date_min = dates[idx_min]
        pid_min = pids[idx_min]
        pid_max = np.max(pids) + 1
        pid_cts = np.zeros(pid_max, dtype=np.int64)
        pid_cts[pid_min] = 1
        uct = 1
        ucts = np.zeros(idx_max, dtype=np.int64)
        ucts[idx_min] = uct
        idx = 1
    
        # For each (date, person)...
        while idx < idx_max:
    
            # Lookup date, person.
            date = dates[idx]
            pid = pids[idx]
    
            # If person count went from 0 to 1, increment unique person count.
            pid_cts[pid] += 1
            if pid_cts[pid] == 1:
                uct += 1
    
            # For past dates outside of window...
            # Note: If window=3, it includes day0,day1,day2.
            while (date - date_min + 1) > window:
    
                # If person count went from 1 to 0, decrement unique person count.
                pid_cts[pid_min] -= 1
                if pid_cts[pid_min] == 0:
                    uct -= 1
                idx_min += 1
                date_min = dates[idx_min]
                pid_min = pids[idx_min]
    
            # Record unique person count.
            ucts[idx] = uct
            idx += 1
    
        return ucts
    

    In [15]:

    # Cast dates to integers.
    df['DateEpoch'] = (df['Date'] - pd.to_datetime('1970-01-01'))/pd.to_timedelta(1, unit='D')
    df['DateEpoch'] = df['DateEpoch'].astype(int)
    

    In [16]:

    %%timeit
    windowed_nunique_corrected(
        dates=df['DateEpoch'].values,
        pids=df['PersonId'].values,
        window=window)
    

    98.8 µs ± 41.3 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

    In [17]:

    # Check accuracy of results.
    test = windowed_nunique_corrected(
        dates=df['DateEpoch'].values,
        pids=df['PersonId'].values,
        window=window)
    assert all(ref == test)
    

答案 1 :(得分:0)

如果您只想要在过去365天内进入建筑物内的唯一身份证号码,您可以先使用.loc在过去365天内限制数据集:

df = df.loc[df['date'] > '2016-09-28',:]

和一个群组你得到的行数与进入的独特人物一样多,如果按照计数进行,你也会得到他们进来的次数:

df = df.groupby('PersonID').count()

这似乎适合你的问题,但也许我弄错了。 祝你有个美好的一天

答案 2 :(得分:0)

非常接近您在种子测试2中的时间,但作为一个衬垫,重新采样超过一年。

 df.resample('AS',on='Date')['PersonId'].expanding(0).apply(lambda x: np.unique(x).shape[0])

时间结果

1 loop, best of 3: 483 ms per loop