如何使用pandas从数据框中获取独特性?

时间:2016-07-11 12:03:56

标签: python pandas

我有df

2016-06-21 06:25:09 upi88@yandex.ru GET HTTP/1.1    Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53   200 application/json    2130    https://edge-chat.facebook.com/pull?channel=p_100006170407238&seq=27&clientid=1d67ca6e&profile=mobile&partition=-2&sticky_token=185&msgs_recv=27&qp=y&cb=1830997782&state=active&sticky_pool=frc3c09_chat-proxy&uid=100006170407238&viewer_uid=100006170407238&m_sess=&__dyn=1Z3p5wnE-4UpwDF3GAgy78qzoC6Erz8B0GxG9xu3Z0QwFzohxO3O2G2a1mwYxm48sxadwpVEy1qK78gwUx6&__req=79&__ajax__=AYlbtcBwGC2suZLI-J88V0PWa58vtQeG3YlQLydFRsAl6UwLSjsSpD7peu8mGl6NsHvd2zxfDcB6A0-XunBugUsYZ1lMYmUu97R43iV7XSfpyg&__user=100006170407238
2016-06-22 06:25:20 upi88@yandex.ru POST HTTP/1.1   Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53   200 application/x-javascript    20248   https://m.facebook.com/stories.php?aftercursor=MTQ2NjY2MzEwNToxNDY2NjYzMTA1Ojg6NzM0ODg0MDExMjAyNDY1MzA5NToxNDY2NjYyNzk1OjA%3D&tab=h_nor&__m_log_async__=1
2016-06-23 06:25:25 upi88@yandex.ru CONNECT HTTP/1.1    Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53   200 -   0   scontent.xx.fbcdn.net:443
2016-06-23 06:25:25 upi88@yandex.ru GET HTTP/1.1    Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53   200 text/html   1105    https://m.facebook.com/xti.php?xt=2.qid.6299270070554694533%3Amf_story_key.343726573953754118%3Aei.AI%40ecf11fb3faf9c0b1f73ce2a74bc9f228
2016-06-24 06:25:25 lemuska@mail.ru CONNECT HTTP/1.1    Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53   200 -   0   scontent.xx.fbcdn.net:443
2016-06-25 06:25:25 lemuska@mail.ru CONNECT HTTP/1.1    Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53   200 -   0   scontent.xx.fbcdn.net:443
2016-06-25 06:25:25 lemuska@mail.ru CONNECT HTTP/1.1    Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53   200 -   0   scontent.xx.fbcdn.net:443

我需要为每个ID(仅年,月和日)获取唯一日期。 期望的输出:

upi88@yandex.ru - 2016-06-21, 2016-06-22, 2016-06-23
lemuska@mail.ru - 2016-06-24, 2016-06-25

我怎样才能得到这个日期?

4 个答案:

答案 0 :(得分:2)

您可以先从日期中提取所需信息:

df['filtered date'] = [w[:10] for w in df['date']]

然后你使用`drop duplicates':

output = df[['id','filtered date']].drop_duplicates()

然后,您可以为了清晰起见对数据框重新排序:

output.sort_values(by['id','filtered date'],inplace = True)

你终于得到了这种输出:

    id               filtered date
0   lemuska@mail.ru  2016-06-24
1   lemuska@mail.ru  2016-06-25
3   upi88@yandex.ru  2016-06-21
4   upi88@yandex.ru  2016-06-22
5   upi88@yandex.ru  2016-06-23

答案 1 :(得分:1)

Pandas为DataFrames提供了groupby函数,它应该适合您的需要。

# Generate dataframe with random values
mail  = ['alice@foo.com', 'bob@bar.com', 'me@baz.com']
stime = datetime.strptime('2016-07-01 00:00:00', '%Y-%m-%d %H:%M:%S')
etime = datetime.strptime('2016-07-30 00:00:00', '%Y-%m-%d %H:%M:%S')
tdelta = etime - stime
tdiff = tdelta.days * 24 * 60 * 60 + tdelta.seconds

df = pd.DataFrame({
    'mail': [choice(mail) for _ in range(10)],
    'time':[stime + timedelta(seconds=randrange(tdiff)) for _ in range(10)]
})

# Group dataframe by column 'mail' and apply the lambda expression to
# transform the grouped set of values into unique time values.
r = df.groupby(by='mail').apply(lambda x: set(x['time'].values))

然后,您应该能够使用结果:

print(r)

mail
alice@foo.com    {2016-07-24T16:42:12.000000000, 2016-07-07T15:...
bob@bar.com      {2016-07-13T18:53:07.000000000, 2016-07-04T06:...
me@baz.com       {2016-07-10T07:37:19.000000000, 2016-07-09T07:...
dtype: object

答案 2 :(得分:1)

这是一个单行(假设dateID作为相关列的名称)

df.groupby('ID').apply(lambda x: (x['date'].str[:10]).unique())

及其输出

ID
lemuska@mail.ru                [2016-06-24, 2016-06-25]
upi88@yandex.ru    [2016-06-21, 2016-06-22, 2016-06-23]
dtype: object

答案 3 :(得分:1)

让我们在下面阅读您的样本数据:

import pandas as pd
import StringIO

df = pd.read_table(StringIO.StringIO("""2016-06-21 06:25:09 upi88@yandex.ru GET HTTP/1.1    Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53   200 application/json    2130    https://edge-chat.facebook.com/pull?channel=p_100006170407238&seq=27&clientid=1d67ca6e&profile=mobile&partition=-2&sticky_token=185&msgs_recv=27&qp=y&cb=1830997782&state=active&sticky_pool=frc3c09_chat-proxy&uid=100006170407238&viewer_uid=100006170407238&m_sess=&__dyn=1Z3p5wnE-4UpwDF3GAgy78qzoC6Erz8B0GxG9xu3Z0QwFzohxO3O2G2a1mwYxm48sxadwpVEy1qK78gwUx6&__req=79&__ajax__=AYlbtcBwGC2suZLI-J88V0PWa58vtQeG3YlQLydFRsAl6UwLSjsSpD7peu8mGl6NsHvd2zxfDcB6A0-XunBugUsYZ1lMYmUu97R43iV7XSfpyg&__user=100006170407238
2016-06-22 06:25:20 upi88@yandex.ru POST HTTP/1.1   Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53   200 application/x-javascript    20248   https://m.facebook.com/stories.php?aftercursor=MTQ2NjY2MzEwNToxNDY2NjYzMTA1Ojg6NzM0ODg0MDExMjAyNDY1MzA5NToxNDY2NjYyNzk1OjA%3D&tab=h_nor&__m_log_async__=1
2016-06-23 06:25:25 upi88@yandex.ru CONNECT HTTP/1.1    Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53   200 -   0   scontent.xx.fbcdn.net:443
2016-06-23 06:25:25 upi88@yandex.ru GET HTTP/1.1    Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53   200 text/html   1105    https://m.facebook.com/xti.php?xt=2.qid.6299270070554694533%3Amf_story_key.343726573953754118%3Aei.AI%40ecf11fb3faf9c0b1f73ce2a74bc9f228
2016-06-24 06:25:25 lemuska@mail.ru CONNECT HTTP/1.1    Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53   200 -   0   scontent.xx.fbcdn.net:443
2016-06-25 06:25:25 lemuska@mail.ru CONNECT HTTP/1.1    Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53   200 -   0   scontent.xx.fbcdn.net:443
2016-06-25 06:25:25 lemuska@mail.ru CONNECT HTTP/1.1    Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53   200 -   0   scontent.xx.fbcdn.net:443
"""), delim_whitespace=True, header=None)

您感兴趣的是第一个(index:0)列,即日期和第三个(索引:2),即电子邮件地址。纯粹出于可见性的原因,让我们将它们隔离在新的数据框中:

df2 = df[[0, 2]]

现在是:

            0                2
0  2016-06-21  upi88@yandex.ru
1  2016-06-22  upi88@yandex.ru
2  2016-06-23  upi88@yandex.ru
3  2016-06-23  upi88@yandex.ru
4  2016-06-24  lemuska@mail.ru
5  2016-06-25  lemuska@mail.ru
6  2016-06-25  lemuska@mail.ru

我们现在需要对它们进行分组并使用自定义函数进行聚合,这会将聚合日期转换为列表(如您所需的输出):

df2.groupby(2).agg(lambda x: x.unique().tolist()).reset_index()

reset_index()修复索引,以便得到以下数据框:

                 2                                     0
0  lemuska@mail.ru              [2016-06-24, 2016-06-25]
1  upi88@yandex.ru  [2016-06-21, 2016-06-22, 2016-06-23]