无法通过drop_down网页从请求中获取数据

时间:2019-12-23 09:51:42

标签: python web-scraping

我从python开始,现在我在BOT网站上使用请求获取数据 我需要贷款利率,但要求它检索个人存款利率中的数据。 Deposit Rates for Individuals

Loan Rates

My code get data from Deposit Rates for Individuals not Loan Rates

My code get data from Deposit Rates for Individuals not Loan Rates

Page Source code

这是我第一次在StackOverflow中工作。我对忽略了该问题表示由衷的歉意,但是我是新用户,而且我没有声誉,因此无法发布完整照片。

1 个答案:

答案 0 :(得分:0)

好吧,因为您没有向我们提供所需的输出。

实际上,您是在向GET请求一个网站,该网站从您那里获取的数据为POST,一旦您点击Loan Rates,就会发生这种情况。

这是一种快速的Selenium方法

from selenium import webdriver
from bs4 import BeautifulSoup
from selenium.webdriver.firefox.options import Options
import pandas as pd

options = Options()
options.add_argument('--headless')

driver = webdriver.Firefox(options=options)
driver.get("https://www.bot.or.th/English/Statistics/FinancialMarkets/Interestrate/_layouts/application/interest_rate/IN_Rate.aspx")
invest = driver.find_element_by_xpath(
    '//*[@id="ctl00_PlaceHolderMain_rdoType_1"]').click()

df = pd.read_html(driver.page_source)

print(df)

您还可以执行以下简单脚本:

import pandas as pd

print("Insert Date In The Following Format\nDD/MM/YYYY")

date = input(">: ")
df = pd.read_csv(
    f"https://www.bot.or.th/English/Statistics/FinancialMarkets/Interestrate/_layouts/application/interest_rate/Download_IN_Rate.aspx?fType=CSV&qDate={date}&qINType=1", delimiter="|")


df.to_csv('out.csv')

您也可以通过POST请求来完成此操作,例如:

from bs4 import BeautifulSoup
import requests
import pandas as pd

data = {
# here you will need to put the POST data, you can retreive that from your browser dev-tools. by location network-tab for the url during POST.
}
r = requests.post(
    "https://www.bot.or.th/English/Statistics/FinancialMarkets/Interestrate/_layouts/application/interest_rate/IN_Rate.aspx", data=data)


soup = BeautifulSoup(r.text, 'html.parser')


for item in soup.findAll("td", {'class': 'tx-news txt-align-left'}):
    print(item.text)