Question

我正在使用API调用来请求数据，然后将返回的数据存储到SQL Server中。我不确定如何将API调用返回的数据与将数据写入SQL Server的函数共享

def req_Data(row,q): 
    """ 
    function to request data from the API
    """

    for record in ds.request_realtime():
        if record.RP_ENTITY_ID in RIC.keys():
            row = [record.data['TIMESTAMP_TZ'],record.RP_STORY_ID,record.RP_ENTITY_ID,record.entity_name,RIC[record.RP_ENTITY_ID], round(record.event_sentiment_score,2),(record.relevance/100)]
        q.put(row)
def write_data(q): 
      row1 = q.get()
      cursor.execute('''INSERT INTO DB()
                       VALUES (?,?,?,?,?,?,?,?,?,?,?)''',row1)
      cnxn.commit()

if __name__ == "__main__": 
    # creating thread 
    row = []
    q = queue.Queue
    t1 = threading.Thread(target=req_Data, name = 'Thread1', args=(row,q)) 
    t2 = threading.Thread(target=write_data,name = 'Thread2', args=(q)) 

    # starting thread 1 
    t1.start() 
    # starting thread 2 

    t2.start() 

    # wait until thread 1 is completely executed 
    t1.join() 
    # wait until thread 2 is completely executed 
    t2.join()

Answer 1

这不是MCVE，但是我会尽力解决它（因为我自己无法测试）。注意事项：

您需要带有括号的q = Queue()才能创建队列对象。
row = []不是必需的，您可以使用如图所示的本地row
使用Queue()从q.task_done()中删除项目您还可以使用q.join()在队列中没有更多项目时继续操作（而不是加入线程，但是您可以这样做）如果需要的话也可以）

考虑到这些因素，它看起来会更像：

import threading
from queue import Queue
import time


def req_Data(q):
    """ function to request data from the API """
    for record in ds.request_realtime():
        if record.RP_ENTITY_ID in RIC.keys():
            row = [record.data['TIMESTAMP_TZ'], record.RP_STORY_ID, record.RP_ENTITY_ID, record.entity_name, RIC[record.RP_ENTITY_ID], round(record.event_sentiment_score, 2), (record.relevance/100)]
            q.put(row)


def write_data(q):
    while True:
        row = q.get()
        cursor.execute('''INSERT INTO DB()
                       VALUES (?,?,?,?,?,?,?,?,?,?,?)''', row)
        cnxn.commit()
        q.task_done()


if __name__ == "__main__":
    # creating thread
    q = Queue() # you were missing the ()
    t1 = threading.Thread(target=req_Data, name='Thread1', args=[q])
    t2 = threading.Thread(target=write_data, name='Thread2', args=[q])

    t1.start()
    time.sleep(10)  # give our queue some time to fill
    t2.start()

    q.join()

但是，如果我要使用多线程，则可能需要多个线程来加载/卸载数据。由于要花费更多的脚本知识来执行此操作以加载数据，因此，我将仅以卸载数据为例。看起来可能像这样：

import threading
from queue import Queue


def req_Data(q):
    """ function to request data from the API """
    for record in ds.request_realtime():
        if record.RP_ENTITY_ID in RIC.keys():
            row = [record.data['TIMESTAMP_TZ'], record.RP_STORY_ID, record.RP_ENTITY_ID, record.entity_name, RIC[record.RP_ENTITY_ID], round(record.event_sentiment_score, 2), (record.relevance/100)]
            q.put(row)


def write_data(q):
    while True:
        row = q.get()
        cursor.execute('''INSERT INTO DB()
                       VALUES (?,?,?,?,?,?,?,?,?,?,?)''', row)
        cnxn.commit()
        q.task_done()


if __name__ == "__main__":
    # creating thread
    q = Queue() # you were missing the ()
    req_Data(q)
    # q is now full
    workers = 10
    thread_list = []
    for i in range(workers):
        t = threading.Thread(target=write_data, args=[q])
        t.start()
        thread_list.append(t)

    q.join()

    for thread in thread_list:
        thread.join()

这是理想的，因为现在调整工作线程数仅需调整workers = 10行即可。使用该脚本，您可能会运行10,000个线程（可能不应该这样做！计算完成后仍要创建线程，这会浪费CPU时间并降低程序速度）

希望这会有所帮助！

尝试使用多线程通过api调用请求数据并将数据存储在数据库中

1 个答案: