Question

我正试图在流媒体视频网站上搜索搜索结果。搜索结果是动态加载的，我认为这就是为什么我没有得到正确的结果。在我提交表单并获得我的results.html后，它总是没有搜索完成的主页...任何帮助都会很棒，如果Mechanize根本没有这个功能也许有人可以指出我在右边方向？提前谢谢。

import mechanize

br = mechanize.Browser()

br.set_handle_robots(False)
br.set_handle_equiv(False)
br.addheaders = [('User-agent', 'Mozilla')]

br.open('http://movietv.to')

br.select_form(nr=0)
br.form.set_all_readonly(False)
br.form.set_value("Godfather", nr=0)

resp = br.submit()

with open('results.html', 'w') as f:
    f.write(resp.read())

Answer 1

Mechanize并不是这个特殊＆＃34;动态＆＃34;现场。最简单的高级方法是通过selenium进行浏览器自动化。

虽然我使用requests和BeautifulSoup HTML解析器使其工作。实现这一目标的关键是1）电影通过XHR请求加载到http://movietv.to/index/loadmovies和2）此请求需要token发送 - 它可以从script中提取主页上的元素。

完整的工作实施：

import re

import requests
from bs4 import BeautifulSoup


search = "Godfather"

token_pattern = re.compile(r'var token_key="(.*?)";')
with requests.Session() as session:  # maintaining web-scraping session
    session.headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"}

    # extract token
    response = session.get("http://movietv.to/")
    soup = BeautifulSoup(response.content, "html.parser")

    script = soup.find("script", text=token_pattern).get_text()
    match = token_pattern.search(script)
    if not match:
        raise ValueError("Cannot find token!")

    token = match.group(1)

    # search for the movies
    response = session.post("http://movietv.to/index/loadmovies", data={
        "loadmovies": "showData",
        "page": "1",
        "abc": "All",
        "genres": "",
        "sortby": "Popularity",
        "quality": "All",
        "type": "movie",
        "q": search,
        "token": token
    })

    soup = BeautifulSoup(response.content, "html.parser")

    for movie in soup.select("div.item"):
        title = movie.find("h2", class_="movie-title")

        print(title.get_text())

打印找到的电影：

The Godfather
The Godfather: Part II
The Godfather: Part III

用机械化问题抓取动态内容

1 个答案: