连接超时Scala JsoupBrowser

时间:2018-09-29 12:17:15

标签: scala parsing web-crawler

有没有一种方法可以在Scala中为JsoupBrowser设置连接超时?它是库https://github.com/ruippeixotog/scala-scraper

1 个答案:

答案 0 :(得分:0)

您需要通过扩展JsoupBrowser基类并覆盖其默认请求设置来创建子类,以对其进行自定义。

import net.ruippeixotog.scalascraper.browser.JsoupBrowser
import org.jsoup.Connection
import scala.concurrent.duration.Duration
import java.util.concurrent.TimeUnit

object Test extends App {

  class CustomJsoupBrowser(timeout: Int) extends JsoupBrowser {
    override protected[this] def defaultRequestSettings(conn: Connection): Connection = {
      super.defaultRequestSettings(conn)
      conn.timeout(timeout)
    }
  }

  val browser = new CustomJsoupBrowser(Duration(10L, TimeUnit.SECONDS).toMillis.toInt)
  val doc = browser.get("http://www.wikipedia.org")
  println(doc)
}