在请求中得到网页的响应

时间:2018-02-27 20:39:07

标签: python web-scraping request lxml

我正在为我的大学网站制作机器人。在该网站中,验证分为两个阶段,第一阶段只有用户名,第二阶段只需要密码。但出于目的,我们可以直接进入密码页面,因为用户名被嵌入到url中。这是链接http://mydy.dypatil.edu/rait/login/index.php?uname=san.yad.rt15@rait.ac.in&wantsurl=

以下是我的代码。

import requests
from lxml import html

session_requests = requests.session()

login_url = "http://mydy.dypatil.edu/rait/login/index.php?uname=san.yad.rt15@rait.ac.in&wantsurl="
result = session_requests.get(login_url)
form_data = {
    "uname_static": "san.yad.rt15@rait.ac.in",
    "username": "san.yad.rt15@rait.ac.in",
    "uname": "san.yad.rt15@rait.ac.in",
    "password": "dypatil@123"
}

result = session_requests.post(
    login_url,
    data=form_data,
    headers=dict(referer=login_url)
)
url = "http://mydy.dypatil.edu/rait/my/"
result = session_requests.get(
    url,
    headers = {"Cache-Control": "no-store, no-cache, must-revalidate, post-check=0, pre-check=0",
    "Connection": "Keep-Alive",
    "Content-Language": "en",
    "Content-Length": "427",
    "Content-Type": "text/html; charset=utf-8",
    "Keep-Alive": "timeout=5, max=98",
    "Location": "http://mydy.dypatil.edu/rait/",
    "Pragma": "no-cache",
    "Server": "Apache/2.4.6 (CentOS) PHP/5.4.16",
    "X-Powered-By": "PHP/5.4.16"}
)
tree = html.fromstring(result.content)
print(tree.text)
# subjects = tree.xpath('//*[@class="launchbutton"/@href]')
#
# for subject in subjects:
#     print(subject)

这不会显示任何错误。此外,当我检查状态代码时,它显示200 Ok。但在回复print(tree.text)中,它显示了几行银行。你能纠正我吗?

1 个答案:

答案 0 :(得分:1)

更改此

print(tree.text)

到这个

print(html.tostring(tree))

获取一些数据:

from lxml import etree

for elem in tree.xpath("//div[@class='name']"):
     print etree.tostring(elem, pretty_print=True)

返回

b'<div class="name">sharad Jadhav</div>\n'
b'<div class="name">Admin User</div>\n'
b'<div class="name">sharad Jadhav</div>\n'