Question

我正在尝试为测试网站编写我的第一个网络抓取工具。它涉及登录，我遵循了有关如何处理此类情况的教程。

import requests
from lxml import html



payload = {
"email": "test_test@test.com",
"password": "123qweasd",
"_token": "3ow4dl7COwnRHa8a6nvNGp4eLkF3wQapT3otGXjR"
 }

rs = requests.session()

login_url = 'https://cloud.webscraper.io/login'
log_page = rs.get(login_url)

tree = html.fromstring(log_page.content)
auth_token = list(set(tree.xpath("//input[@name='_token']/@value")))[0]

login = rs.post(login_url,data=payload, headers=dict(referer=login_url))

url = "https://cloud.webscraper.io/sitemaps"
result = rs.get(url, headers=dict(referer=url))

tree = html.fromstring(result.text)
sidebar_cat = tree.xpath('//*[@id="main-menu-inner"]/ul')

print(sidebar_cat)

我希望该脚本从边栏中列出类别。似乎该脚本每次都会返回并清空列表。当前输出为

"[] 
Process finished with exit code 0"

Answer 1

您已经提取了_token值，但改用了硬编码值。尝试将提取的值传递给payload：

import requests
from lxml import html

rs = requests.session()

login_url = 'https://cloud.webscraper.io/login'
log_page = rs.get(login_url)

tree = html.fromstring(log_page.content)
auth_token = tree.xpath("//input[@name='_token']/@value")[0]

payload = {
    "email": "test_test@test.com",
    "password": "123qweasd",
    "_token": auth_token
 }

login = rs.post(login_url,data=payload, headers=dict(referer=login_url))

url = "https://cloud.webscraper.io/sitemaps"
result = rs.get(url, headers=dict(referer=url))

tree = html.fromstring(result.text)
sidebar_cat = tree.xpath('//*[@id="main-menu-inner"]/ul')

print(sidebar_cat)

Answer 2

尝试打印出“结果”，然后暂时注释掉所有内容。这样，您可以知道请求部分是否出错，或者问题是否来自响应处理。如果响应按预期打印，则对最后三行代码进行故障排除。如果不是，则对代码的请求部分进行故障排除。

Web抓取脚本返回和空列表

2 个答案: