我对Python比较陌生。以该站点为例,我试图抓取餐厅的信息,但是不确定垂直读取数据时如何水平旋转数据。我希望Excel工作表具有六列,如下所示:名称,街道,城市,州,邮政编码,电话。这是我正在使用的代码:
from selenium import webdriver
from bs4 import BeautifulSoup
from urllib.request import urlopen
import time
driver = webdriver.Chrome(executable_path=r"C:\Downloads\chromedriver_win32\chromedriver.exe")
driver.get('https://www.restaurant.com/listing?&&st=KS&p=KS&p=PA&page=1&&searchradius=50&loc=10021')
time.sleep(10)
with urlopen(driver.current_url) as response:
soup = BeautifulSoup(response, 'html.parser')
pageList = soup.findAll("div", attrs={"class": {"details"}})
list_of_inner_text = [x.text for x in pageList]
text = ', '.join(list_of_inner_text)
print(text)
谢谢
编辑:根据反馈,这是我对本页前五家餐厅的期望: FirstFiveRestaurants
答案 0 :(得分:0)
这是一种方法。您的里程可能在其他页面上有所不同。
此行
details = [re.sub(r'\s{2,}|[,]', '',i) for i in restuarant.select_one('h3 + p').text.strip().split('\n') if i!=''
通过拆分'\ n'上的p
标签并进行一些字符串清理,基本上可以处理输出列(栏名)的生成。
import requests, re
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd
driver = webdriver.Chrome(executable_path=r"C:\Users\User\Documents\chromedriver.exe")
driver.get('https://www.restaurant.com/listing?&&st=KS&p=KS&p=PA&page=1&&searchradius=50&loc=10021')
WebDriverWait(driver,10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".restaurants")))
soup = BeautifulSoup(driver.page_source, 'html.parser')
restuarants = soup.select('.restaurants')
results = []
for restuarant in restuarants:
details = [re.sub(r'\s{2,}|[,]', '',i) for i in restuarant.select_one('h3 + p').text.strip().split('\n') if i!='']
details.insert(0, restuarant.select_one('h3 a').text)
results.append(details)
df = pd.DataFrame(results, columns= ['Name','Address', 'City', 'State', 'Zip', 'Phone'])
df.to_csv(r'C:\Users\User\Desktop\Data.csv', sep=',', encoding='utf-8-sig',index = False )