Question

我的目标是从PGA网站获取数据，以提取美国所有高尔夫球场的位置。我的目标是从907页中删除姓名，地址，所有权，电话号码和网站。

我在下面创建了脚本但是在创建CSV时会产生错误。从脚本创建的CSV文件具有前几页和网站页面的数据重复。它没有提供907页的全部数据。

如何修复我的脚本，以便它会刮掉所有907页并生成包含PGA网站上列出的所有高尔夫球场的CSV？

以下是我的剧本：

import csv
import requests 
from bs4 import BeautifulSoup

for i in range(907):      # Number of pages plus one 
     url = "http://www.pga.com/golf-courses/search?page={}&searchbox=Course+Name&searchbox_zip=ZIP&distance=50&price_range=0&course_type=both&has_events=0".format(i)
     r = requests.get(url)
     soup = BeautifulSoup(r.content)
g_data2=soup.find_all("div",{"class":"views-field-nothing"})

courses_list=[]

for item in g_data2:
     try:
          name=item.contents[1].find_all("div",{"class":"views-field-title"})[0].text
     except:
          name=''
     try:
          address1=item.contents[1].find_all("div",{"class":"views-field-address"})[0].text
     except:
          address1=''
     try:
          address2=item.contents[1].find_all("div",{"class":"views-field-city-state-zip"})[0].text
     except:
          address2=''
     try:
          website=item.contents[1].find_all("div",{"class":"views-field-website"})[0].text
     except:
          website=''   
     try:
          Phonenumber=item.contents[1].find_all("div",{"class":"views-field-work-phone"})[0].text
     except:
          Phonenumber=''      

     course=[name,address1,address2,website,Phonenumber]
     courses_list.append(course)

     with open ('PGA_Data.csv','a') as file:
          writer=csv.writer(file)
          for row in courses_list:
               writer.writerow(row)

Answer 1

她是你想要的代码。它将首先在进入下一个之前解析当前页面。（有一些空行，我希望你能自己解决这个问题。）

import csv import requests from bs4 import BeautifulSoup def encode(l): out = [] for i in l: text = str(i).encode('utf-8') out.append(''.join([i if ord(i) < 128 else ' ' for i in text])) #taken from Martjin Pieter's answer # http://stackoverflow.com/questions/20078816/replace-non-ascii-characters-with-a-single-space/20078869#20078869 return out courses_list = [] for i in range(5): # Number of pages plus one url = "http://www.pga.com/golf-courses/search?page={}&searchbox=Course+Name&searchbox_zip=ZIP&distance=50&price_range=0&course_type=both&has_events=0".format(i) r = requests.get(url) soup = BeautifulSoup(r.content) g_data2=soup.find_all("div",{"class":"views-field-nothing"}) for item in g_data2: try: name = item.contents[1].find_all("div",{"class":"views-field-title"})[0].text except: name='' try: address1= item.contents[1].find_all("div",{"class":"views-field-address"})[0].text except: address1='' try: address2= item.contents[1].find_all("div",{"class":"views-field-city-state-zip"})[0].text except: address2='' try: website= item.contents[1].find_all("div",{"class":"views-field-website"})[0].text except: website='' try: Phonenumber= item.contents[1].find_all("div",{"class":"views-field-work-phone"})[0].text except: Phonenumber='' course=[name,address1,address2,website,Phonenumber] courses_list.append(encode(course)) with open ('PGA_Data.csv','a') as file: writer=csv.writer(file) for row in courses_list: writer.writerow(row)

编辑：在unicode编码/解码不可避免的问题之后，我已经修改了答案，它（希望）现在可以工作了。但是http://nedbatchelder.com/text/unipain.html看到了这一点。

循环不用于使用python和beautifulsoup4抓取数据

1 个答案: