Question

如何在Scala中使用StandaloneAhcWSClient发出大量简单的get请求？（这是Play2框架随附的默认http客户端）。就我而言，我收到了约10万个GET请求，以向外部API发送请求。 Future.traverse()不会削减它，是否有更好的方法，也许是一种像处理流一样处理url列表的方法？

这是我现在拥有的代码：https://scastie.scala-lang.org/HgrIyR23TmG12j3MzMCxUw

它最多可以处理列表中的一定数量的url，但是会大量中断，例外情况是java.lang.IllegalStateException: failed to create a child event loop

Answer 1

这就是我最终得到的：

scalaVersion := "2.11.12"

val liftVersion = "2.6"

libraryDependencies ++= Seq(
  "com.typesafe.play" %% "play-ahc-ws-standalone" % "2.0.3",
  "com.typesafe.play" %% "play-ws-standalone-json" % "2.0.3",
  "com.typesafe.play" %% "play-json" % "2.7.2"
)

使用以下build.sbt：

from bs4 import BeautifulSoup
import requests
from twilio.rest import Client

account_sid = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
auth_token = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
twilio_phone_number = '+12345678900'
my_phone_number = '+12345678900'

url = 'https://spokane.craigslist.org/search/sss?query=s10&sort=rel'
webpage = requests.get(url)
soup = BeautifulSoup(webpage.text, 'html.parser')
body = []

datas = soup.find_all(class_ = "result-title hdrlnk", limit = 2)
for data in datas:
        body = 'New S10 Posts!\n' + '\n'.join(data.text)
        client = Client(account_sid, auth_token)
        client.messages.create (
                body = body,
                to = my_phone_number,
                from_ = twilio_phone_number
        )

如何使用StandaloneAhcWSClient

1 个答案: