Question

我想每4秒钟爬一次一个网站，我该怎么做。我的代码在下面。

import requests 
from bs4 import BeautifulSoup 

site = requests.get("http://example.com") 
soup =BeautifulSoup(site.text,'html.parser')
r = str(soup).split(",")
update_time = r[0]
price1 = r[2]
price2 = r[3]
print(update_time,price1,price2)

Answer 1

您可以使用time和threading模块

import requests 
from threading import Thread
from time import sleep
from bs4 import BeautifulSoup 

def scrape():
    site = requests.get("http://example.com") 
    soup =BeautifulSoup(site.text,'html.parser')
    r = str(soup).split(",")
    update_time = r[0]
    price1 = r[2]
    price2 = r[3]
    print(update_time,price1,price2)

for i in range(14400):
    t = Thread(target=scrape)
    t.start()
    sleep(1)

Answer 2

您可以为此使用计划模块。

import schedule
import time
import requests 
from bs4 import BeautifulSoup 

def crawl():
    site = requests.get("http://example.com") 
    soup =BeautifulSoup(site.text,'html.parser')
    r = str(soup).split(",")
    update_time = r[0]
    price1 = r[2]
    price2 = r[3]
    print(update_time,price1,price2)

schedule.every(1).seconds.do(crawl)

while True:
    schedule.run_pending()
    time.sleep(1)

四个小时的窗口可以通过crontab或for循环来实现。

您必须安装schedule模块才能运行上述脚本

sudo pip install schedule

每秒发送请求到网站进行爬网

2 个答案: