Question

我试图了解多处理和池来处理MySQL数据库中的一些推文。这是代码和错误消息。

import multiprocessing
import sqlalchemy
import pandas as pd
import config
from nltk import tokenize as token

q = multiprocessing.Queue()
engine = sqlalchemy.create_engine(config.sqlConnectionString)


def getRow(pandasSeries):
    df = pd.DataFrame()
    tweetTokenizer = token.TweetTokenizer()
    print(pandasSeries.loc['BODY'], "\n", type(pandasSeries.loc['BODY']))
    for tokens in tweetTokenizer.tokenize(pandasSeries.loc['BODY']):
        df = df.append(pd.Series(data=[pandasSeries.loc['ID'], tokens, pandasSeries.loc['AUTHOR'],
                                  pandasSeries.loc['RETWEET_COUNT'], pandasSeries.loc['FAVORITE_COUNT'],
                                  pandasSeries.loc['FOLLOWERS_COUNT'], pandasSeries.loc['FRIENDS_COUNT'],
                                  pandasSeries.loc['PUBLISHED_AT']],
                                 index=['id', 'tweet', 'author', 'retweet', 'fav', 'followers', 'friends',
                                        'published_at']), ignore_index=True)
    df.to_sql(name="tweet_tokens", con=engine, if_exists='append')


if __name__ == '__main__':
    ##LOADING SQL INTO DATAFRAME##
    databaseData = pd.read_sql_table(config.tweetTableName, engine)

    pool = multiprocessing.Pool(6)
    for row in databaseData.iterrows():
        print(row)
        pool.map(getRow, row)
    pool.close()
    q.close()
    q.join_thread()


"""
            OUPUT

C:\Users\Def\Anaconda3\python.exe C:/Users/Def/Dropbox/Dissertation/testThreadCopy.py
(0, ID                                                              3247
AUTHOR                                             b'Elon Musk News'
RETWEET_COUNT                                                      0
FAVORITE_COUNT                                                     0
FOLLOWERS_COUNT                                                20467
FRIENDS_COUNT                                                  14313
BODY               Elon Musk Takes an Adorable 5th Grader's Idea ...
PUBLISHED_AT                                     2017-03-03 00:00:01
Name: 0, dtype: object)
Elon Musk Takes an Adorable 5th Grader's 
 <class 'str'>
multiprocessing.pool.RemoteTraceback: 

Traceback (most recent call last):
  File "C:\Users\Def\Anaconda3\lib\multiprocessing\pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "C:\Users\Def\Anaconda3\lib\multiprocessing\pool.py", line 44, in mapstar
    return list(map(*args))
  File "C:\Users\Def\Dropbox\Dissertation\testThreadCopy.py", line 16, in getRow
    print(pandasSeries.loc['BODY'], "\n", type(pandasSeries.loc['BODY']))
AttributeError: 'numpy.int64' object has no attribute 'loc'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:/Users/Def/Dropbox/Dissertation/testThreadCopy.py", line 34, in <module>
    pool.map(getRow, row)
  File "C:\Users\Def\Anaconda3\lib\multiprocessing\pool.py", line 260, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Users\Def\Anaconda3\lib\multiprocessing\pool.py", line 608, in get
    raise self._value
AttributeError: 'numpy.int64' object has no attribute 'loc'

Process finished with exit code 1
"""

我不明白为什么它打印出第一个系列然后崩溃？为什么说pandasSeries.loc [＆＃39; BODY＆＃39;]的类型为numpy.int64，当打印出来时它表示它是字符串类型？我确定我在其他许多地方出了问题，如果你能看到哪里，请你指出来。感谢。

Answer 1

当我构建一个简单的数据帧时：

frame
   0  1   2   3
0  0  1   2   3
1  4  5   6   7
2  8  9  10  11

并迭代两次我得到：

for row in databaseData.iterrows():
    for i in row:
        print(i, type(i))

该内部循环产生2个项目，一个行索引/标签，以及一个带有值的系列。

0 <class 'numpy.int64'>
0    0
1    1
2    2
3    3
Name: 0, dtype: int32 <class 'pandas.core.series.Series'>

你的map做同样的事情，向一个进程发送一个数字索引（产生错误），一个系列发送给另一个进程。

如果我使用pool.map而不使用for row：

pool.map(getRow, databaseData.iterrows())

然后getRow收到一个2元素元组。

def getRow(aTuple):
    rowlbl, rowSeries = aTuple
    print(rowSeries)   
    ...

你的print(row)显示了这个元组;它更难看，因为系列部分是多线的。如果我添加\ n它可能更清楚

(0,                                   # row label
ID                               3247       # multiline Series
AUTHOR                           b'Elon Musk News'
RETWEET_COUNT                    0
....
Name: 0, dtype: object)

Python 3.5多处理池＆amp;＆amp; ＆＃39; numpy.int64没有属性.loc＆＃39;

1 个答案: