熊猫-返回日期范围内的单个日期并匹配工作日的二进制值

Question

数据集：

下面的数据集应该复制旅行公司的时间表数据集（例如，火车，公共汽车或飞机等路线）

df = pd.DataFrame({'operator': ['op_a', 'op_a', 'op_a', 'op_a', 'op_b', 'op_b', 'op_b', 'op_b', 'op_c', 'op_c', 'op_c', 'op_c', 'op_d', 'op_d'],
                   'from': ['a', 'a', 'a', 'a', 'c', 'c', 'c', 'c', 'a', 'a', 'a', 'a', 'x', 'x'], 
                   'to': ['b', 'b', 'b', 'b', 'd', 'd', 'd', 'd', 'b', 'b', 'b', 'b', 'y', 'y'], 
                   'valid_from': ['13/11/2018', '13/11/2018', '13/11/2018', '13/11/2018', '13/11/2018', '13/11/2018', '13/11/2018', '13/11/2018', '15/02/2019', '15/02/2019', '15/02/2019', '15/02/2019', '20/05/2019', '21/05/2019'],
                   'valid_to': ['20/11/2018', '20/11/2018', '19/11/2018', '19/11/2018', '19/11/2018', '19/11/2018', '21/11/2018', '21/11/2018', '21/02/2019', '21/02/2019', '20/02/2019', '20/02/2019', '30/05/2019', '29/05/2019'], 
                   'day_of_week': ['0101010', '0100010', '0111100', '1101100', '0101010', '0100010', '0111100', '1101100', '0001101', '1110000', '0000000', '0000001', '1000000', '1000001']})
    print(df)

operator-运营公司，例如ABC航空公司，DEF火车公司

from-例如伦敦，纽约，纳尼亚

to-目的地，例如巴黎

valid_from-日期范围的开始（可以是一周中的任何一天），运营商可以购买该路线，例如2019-11-01

valid_to-日期范围的结束日期（可以是一周中的任何一天），可供运营商购买路线，例如2019-11-12

day_of_week-二进制，表示Sun到星期六的可用性，例如0101010表示该路线在日期范围内的周一，周三和周五可用

必填：

将日期范围转换为单个日期及其从day_of_week字段获得的可用性的输出数据集。主要目标是获得一个干净的数据集，然后将其加载到Tableau中，然后生成一个可以轻松显示路线可用性的报告。

所需的输出：

dfout = pd.DataFrame({'operator': ['op_a', 'op_a', 'op_a', 'op_a', 'op_a', 'op_a', 'op_a'], 'from': ['a', 'a', 'a', 'a', 'a', 'a', 'a'], 'to': ['b', 'b', 'b', 'b', 'b', 'b', 'b'], 'date': ['13/11/2018', '14/11/2018', '15/11/2018', '16/11/2018', '17/11/2018', '18/11/2018', '19/11/2018'], 'available': [1, 1, 1, 1, 0, 1, 1]})
print(dfout)

因此这将是日期范围op_a至a的路线b至2018-11-13的{​​{1}}的输出。

数据集很奇怪。日期范围可以是非常随机的，但是2018-11-19将始终显示该日期范围内一周中的几天的可用性。某些相同的日期范围甚至可能具有不同的day_of_week二进制组合，但是本质上，如果day_of_week在任何时候都指示了给定日期范围，路线和运营商的可用性，则将其视为日期可用。

我尝试做的事情：

使用以下帮助：Pandas: decompress date range to individual dates

day_of_week

结果看起来很有希望。我最后的想法是：

1. 添加一个import pandas as pd df = pd.DataFrame({'operator': ['op_a', 'op_a', 'op_a', 'op_a', 'op_b', 'op_b', 'op_b', 'op_b', 'op_c', 'op_c', 'op_c', 'op_c', 'op_d', 'op_d'], 'from': ['a', 'a', 'a', 'a', 'c', 'c', 'c', 'c', 'a', 'a', 'a', 'a', 'x', 'x'], 'to': ['b', 'b', 'b', 'b', 'd', 'd', 'd', 'd', 'b', 'b', 'b', 'b', 'y', 'y'], 'valid_from': ['13/11/2018', '13/11/2018', '13/11/2018', '13/11/2018', '13/11/2018', '13/11/2018', '13/11/2018', '13/11/2018', '15/02/2019', '15/02/2019', '15/02/2019', '15/02/2019', '20/05/2019', '21/05/2019'], 'valid_to': ['20/11/2018', '20/11/2018', '19/11/2018', '19/11/2018', '19/11/2018', '19/11/2018', '21/11/2018', '21/11/2018', '21/02/2019', '21/02/2019', '20/02/2019', '20/02/2019', '30/05/2019', '29/05/2019'], 'day_of_week': ['0101010', '0100010', '0111100', '1101100', '0101010', '0100010', '0111100', '1101100', '0001101', '1110000', '0000000', '0000001', '1000000', '1000001']}) df.set_index(['operator', 'from','to'], inplace=True) df['valid_from'] = pd.to_datetime(df['valid_from']) df['valid_to'] = pd.to_datetime(df['valid_to']) df['row'] = range(len(df)) starts = df[['valid_from', 'day_of_week', 'row']].rename(columns={'valid_from': 'date'}) ends = df[['valid_to', 'day_of_week', 'row']].rename(columns={'valid_to':'date'}) df_decomp = pd.concat([starts, ends]) df_decomp = df_decomp.set_index('row', append=True) df_decomp.sort_index() df_decomp = df_decomp.groupby(level=[0,1,2,3]).apply(lambda x: x.set_index('date').resample('D').fillna(method='pad')) 列，该列返回weekday以date开头的Sunday的工作日
2. 添加一个0列，以available作为位置索引返回day_of_week中的二进制值
3. 最后，要以某种方式删除重复的weekday，operator和from行，并保留具有to的{​​{1}}行，并删除{{ 1}}，或者如果没有available / 1 / 0的{​​{1}}，则将可用值保留为1。 ..

疯狂...为冗长的事情道歉，我希望我有道理。任何帮助，将不胜感激。

修改：

• 更新了上面的“我尝试做的事情”部分。
• 更新了数据集，以便在日期中添加更多的变化（仍然是刚刚调整过operators个日期的相同数据集）

Answer 1

如果您不太关心速度，可以使用iterrows（）和df.at []：

import pandas as pd

df = pd.DataFrame({'operator': ['op_a', 'op_a', 'op_a', 'op_a', 'op_b', 'op_b', 'op_b', 'op_b', 'op_c', 'op_c', 'op_c', 'op_c', 'op_d', 'op_d'], 'from': ['a', 'a', 'a', 'a', 'c', 'c', 'c', 'c', 'a', 'a', 'a', 'a', 'x', 'x'], 'to': ['b', 'b', 'b', 'b', 'd', 'd', 'd', 'd', 'b', 'b', 'b', 'b', 'y', 'y'], 'valid_from': ['13/11/2018', '13/11/2018', '13/11/2018', '13/11/2018', '13/11/2018', '13/11/2018', '13/11/2018', '13/11/2018', '15/02/2019', '15/02/2019', '15/02/2019', '15/02/2019', '01/05/2019', '01/05/2019'], 'valid_to': ['19/11/2018', '19/11/2018', '19/11/2018', '19/11/2018', '19/11/2018', '19/11/2018', '19/11/2018', '19/11/2018', '21/02/2019', '21/02/2019', '21/02/2019', '21/02/2019', '10/05/2019', '11/05/2019'], 'day_of_week': ['0101010', '0100010', '0111100', '1101100', '0101010', '0100010', '0111100', '1101100', '0001101', '1110000', '0000000', '0000001', '1000000', '1000001']})

df['valid_from'] = pd.to_datetime(df['valid_from'])
df['valid_to'] = pd.to_datetime(df['valid_to'])
df['day'] = (df['valid_from']+pd.to_timedelta(1, unit='d')).dt.weekday # gives weekdays : ) = Sunday
print df.head()


df_out = pd.DataFrame(columns=['available', 'date', 'from', 'operator', 'to'])

idx = 0
for i, row in df.iterrows():
    daterange = row['valid_to'] - row['valid_from']
    print daterange.days

    daystring = 52 * (row['day_of_week'])  # extend string to allow going through multiple weeks

    for j in range(daterange.days+1):
        df_out.at[idx, ['available', 'date', 'from', 'operator', 'to']] = [ # replaced set_value with df.at[]
            int(daystring[j + row['day']]), # use day of the week as starting position
            row['valid_from']+pd.to_timedelta(j, unit='d'),
            row['from'],
            row['operator'],
            row['to']
            ]

        # row['day_of_week'][j]
        idx += 1

df_out.drop_duplicates(inplace=True) # drop all duplicates
df_0 = df_out[df_out['available']==0]
df_1 = df_out[df_out['available']==1]
df_out = df_0.merge(df_1, how='outer', left_on=['date', 'from', 'operator', 'to'], right_on=['date', 'from', 'operator', 'to'])
df_out.fillna(0, inplace=True)

df_out['available'] = df_out['available_x'] + df_out['available_y']
df_out.drop(['available_x', 'available_y'], axis=1, inplace=True)
df_out.sort_values(by='date',inplace=True)
print df_out

Answer 2

这可以解决问题：

import pandas as pd
import numpy as np

# dataset
df = pd.DataFrame({'operator': ['op_a', 'op_a', 'op_a', 'op_a', 'op_b', 'op_b', 'op_b', 'op_b', 'op_c', 'op_c', 'op_c', 'op_c', 'op_d', 'op_d'],
                   'from': ['a', 'a', 'a', 'a', 'c', 'c', 'c', 'c', 'a', 'a', 'a', 'a', 'x', 'x'], 
                   'to': ['b', 'b', 'b', 'b', 'd', 'd', 'd', 'd', 'b', 'b', 'b', 'b', 'y', 'y'], 
                   'valid_from': ['13/11/2018', '13/11/2018', '13/11/2018', '13/11/2018', '13/11/2018', '13/11/2018', '13/11/2018', '13/11/2018', '15/02/2019', '15/02/2019', '15/02/2019', '15/02/2019', '20/05/2019', '21/05/2019'],
                   'valid_to': ['20/11/2018', '20/11/2018', '19/11/2018', '19/11/2018', '19/11/2018', '19/11/2018', '21/11/2018', '21/11/2018', '21/02/2019', '21/02/2019', '20/02/2019', '20/02/2019', '30/05/2019', '29/05/2019'], 
                   'day_of_week': ['0101010', '0100010', '0111100', '1101100', '0101010', '0100010', '0111100', '1101100', '0001101', '1110000', '0000000', '0000001', '1000000', '1000001']})

# set operator, from, to as index
df.set_index(['operator', 'from','to'], inplace=True)

# convert date ranges to datetime types
df['valid_from'] = pd.to_datetime(df['valid_from'])
df['valid_to'] = pd.to_datetime(df['valid_to'])

# bring individual dates in date ranges and stack
df['row'] = range(len(df))
starts = df[['valid_from', 'day_of_week', 'row']].rename(columns={'valid_from': 'date'})
ends = df[['valid_to', 'day_of_week', 'row']].rename(columns={'valid_to':'date'})

df_decomp = pd.concat([starts, ends])
df_decomp = df_decomp.set_index('row', append=True)
df_decomp.sort_index()

df_decomp = df_decomp.groupby(level=[0,1,2,3]).apply(lambda x: x.set_index('date').resample('D').fillna(method='pad'))

# remove indexes
df_decomp.reset_index(level=3, drop=True, inplace=True)
df_decomp.reset_index(inplace=True)

# create weekday column
df_decomp['weekday'] = np.where(df_decomp['date'].dt.weekday == 6, 
                            df_decomp['date'].dt.weekday - 6, 
                            df_decomp['date'].dt.weekday + 1)

# use weekday to extract availability in day_of_week
df_decomp['available'] = [b[a] for a, b in zip(df_decomp['weekday'], df_decomp['day_of_week'])]
df_decomp['available'] = df_decomp['available'].astype('int')

# sort values and remove duplicates with available=1 taking priority
df_decomp = df_decomp.sort_values('available', ascending=False).drop_duplicates(['operator','from','to','date'])
df_decomp = df_decomp.sort_values(['operator','from','to','date'])

df_decomp