Question

我有一段时间没有完成扭曲的编程，所以我正试图重新进入一个新项目。我正在尝试设置一个可以将服务器列表作为参数的扭曲客户端，并且对于每个服务器，它发送API GET调用并将返回消息写入文件。此API GET调用应每60秒重复一次。

我使用Twisted的代理类在单个服务器上成功完成了它：

from StringIO import StringIO

from twisted.internet import reactor
from twisted.internet.protocol import Protocol
from twisted.web.client import Agent
from twisted.web.http_headers import Headers
from twisted.internet.defer import Deferred

import datetime
from datetime import timedelta
import time

count = 1
filename = "test.csv"

class server_response(Protocol):
    def __init__(self, finished):
        print "init server response"
        self.finished = finished
        self.remaining = 1024 * 10

    def dataReceived(self, bytes):
        if self.remaining:
            display = bytes[:self.remaining]
            print 'Some data received:'
            print display
            with open(filename, "a") as myfile:
                myfile.write(display)

            self.remaining -= len(display)


    def connectionLost(self, reason):
        print 'Finished receiving body:', reason.getErrorMessage()

        self.finished.callback(None)

def capture_response(response): 
    print "Capturing response"
    finished = Deferred()
    response.deliverBody(server_response(finished))
    print "Done capturing:", finished

    return finished

def responseFail(err):
    print "error" + err
    reactor.stop()


def cl(ignored):
    print "sending req"
    agent = Agent(reactor)
    headers = {
    'authorization': [<snipped>],
    'cache-control': [<snipped>],
    'postman-token': [<snipped>]
    }

    URL = <snipped>
    print URL

    a = agent.request(
        'GET',
        URL,
        Headers(headers),
        None)

    a.addCallback(capture_response)
    reactor.callLater(60, cl, None)
    #a.addBoth(cbShutdown, count)


def cbShutdown(ignored, count):
    print "reactor stop"
    reactor.stop()

def parse_args():
    usage = """usage: %prog [options] [hostname]:port ...
    Run it like this:
      python test.py hostname1:instanceName1 hostname2:instancename2 ...
    """

    parser = optparse.OptionParser(usage)

    _, addresses = parser.parse_args()

    if not addresses:
        print parser.format_help()
        parser.exit()

    def parse_address(addr):
        if ':' not in addr:
            hostName = '127.0.0.1'
            instanceName = addr
        else:
            hostName, instanceName = addr.split(':', 1)

        return hostName, instanceName

    return map(parse_address, addresses)

if __name__ == '__main__':
    d = Deferred()
    d.addCallbacks(cl, responseFail)
    reactor.callWhenRunning(d.callback, None)

    reactor.run()

但是，我很难搞清楚如何让多个座席发送电话。有了这个，我依靠cl（）--- reactor.callLater（60，cl，None）中的写入结束来创建调用循环。 那么我如何创建多个呼叫代理协议（server_response（协议））并在我的反应堆启动后继续为每个协议循环执行GET？

Answer 1

看看那只猫拖进去了！

那么我该如何创建多个呼叫代理

使用treq。你很少想与private EndDrawerToggle drawerToggle; ... public void initNavigationDrawer() { NavigationView navigationView = ... ... drawerLayout = (DrawerLayout)findViewById(R.id.drawer); drawerToggle = new EndDrawerToggle(this, drawerLayout, toolbar, R.string.drawer_open, R.string.drawer_close); drawerLayout.addDrawerListener(drawerToggle); } @Override protected void onPostCreate(Bundle savedInstanceState) { super.onPostCreate(savedInstanceState); drawerToggle.syncState(); }类纠缠在一起。

此API GET调用应每60秒重复一次

使用LoopingCalls代替Agent，在这种情况下，它会更容易，以后您会遇到更少的问题。

callLater

它从import treq from twisted.internet import task, reactor filename = 'test.csv' def writeToFile(content): with open(filename, 'ab') as f: f.write(content) def everyMinute(*urls): for url in urls: d = treq.get(url) d.addCallback(treq.content) d.addCallback(writeToFile) #----- Main -----# sites = [ 'https://www.google.com', 'https://www.amazon.com', 'https://www.facebook.com'] repeating = task.LoopingCall(everyMinute, *sites) repeating.start(60) reactor.run()函数开始，每60秒运行一次。在该函数中，每个端点都被查询，一旦响应的内容变得可用，everyMinute()函数将获取响应并返回内容。最后，内容被写入文件。

PS

您是在抓取还是试图从这些网站中提取某些内容？如果您scrapy可能是一个不错的选择。

编写Twisted Client以向多个API调用发送循环GET请求并记录响应

1 个答案: