我怎样才能将报废的数据水平导出到Excel?

时间:2019-08-27 15:16:31

标签: python web-scraping beautifulsoup

我对Python比较陌生。以该站点为例,我试图抓取餐厅的信息,但是不确定垂直读取数据时如何水平旋转数据。我希望Excel工作表具有六列,如下所示:名称,街道,城市,州,邮政编码,电话。这是我正在使用的代码:

from selenium import webdriver
from bs4 import BeautifulSoup
from urllib.request import urlopen
import time

driver = webdriver.Chrome(executable_path=r"C:\Downloads\chromedriver_win32\chromedriver.exe")


driver.get('https://www.restaurant.com/listing?&&st=KS&p=KS&p=PA&page=1&&searchradius=50&loc=10021')
time.sleep(10)
with urlopen(driver.current_url) as response:
    soup = BeautifulSoup(response, 'html.parser')
    pageList = soup.findAll("div", attrs={"class": {"details"}})
    list_of_inner_text = [x.text for x in pageList]
    text = ', '.join(list_of_inner_text)
    print(text)

谢谢

编辑:根据反馈,这是我对本页前五家餐厅的期望: FirstFiveRestaurants

1 个答案:

答案 0 :(得分:0)

这是一种方法。您的里程可能在其他页面上有所不同。

此行

details = [re.sub(r'\s{2,}|[,]', '',i) for i in restuarant.select_one('h3 + p').text.strip().split('\n') if i!=''

通过拆分'\ n'上的p标签并进行一些字符串清理,基本上可以处理输出列(栏名)的生成。

import requests, re
from bs4 import BeautifulSoup 
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd

driver = webdriver.Chrome(executable_path=r"C:\Users\User\Documents\chromedriver.exe")
driver.get('https://www.restaurant.com/listing?&&st=KS&p=KS&p=PA&page=1&&searchradius=50&loc=10021')
WebDriverWait(driver,10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".restaurants")))
soup = BeautifulSoup(driver.page_source, 'html.parser')
restuarants = soup.select('.restaurants')
results = []

for restuarant in restuarants:
    details = [re.sub(r'\s{2,}|[,]', '',i) for i in restuarant.select_one('h3 + p').text.strip().split('\n') if i!='']
    details.insert(0, restuarant.select_one('h3 a').text)
    results.append(details)

df = pd.DataFrame(results, columns= ['Name','Address', 'City', 'State', 'Zip', 'Phone'])
df.to_csv(r'C:\Users\User\Desktop\Data.csv', sep=',', encoding='utf-8-sig',index = False )

enter image description here