如何将2个Webscraped列表合并为一个

时间:2018-05-21 01:36:42

标签: python html python-3.x web-scraping jupyter

我尝试将两个Web抓取列表合并到一个列表中,但它只是显示一个实例。 (我已经有了名单和地址列表,我只想加入它们。)

/fonts/vendor/ionicons/dist/

第一份清单:

from bs4 import BeautifulSoup
import urllib.request
def get_HTML(url):
    response = urllib.request.urlopen(url)
    html = response.read()
    return html

输出:

venues_html = get_HTML('http://www.cxra.com/venues/new-york/')
soup = BeautifulSoup(venues_html, "lxml")
for venue in soup('a', attrs={'href' : '#', 'onclick' : 'return false;'}):
    display (venue.text)

第二个清单

'Manhattan Center Studios'
'Ellis Island'
'The TimesCenter'
'The Altman Building'
'NYIT Auditorium on Broadway'

输出:

for info in soup.findAll('div', attrs={'class' : 'infoUnit col-md-6'}):
    display (info.text)

尝试加入两者:

'\n \n311 West 34th Street\r\nNew York City, 94710\n\n212.613.5536\n'
'\n \nEllis Island\r\nNew York, NY 10004\n\n212.613.5535\n'
'\n \n242 W 41st St\r\nNew York, NY 10036\n\n212.613.5535\n'
'\n \n135 W 18th St\r\nNew York, NY 10011\n\n212.613.5535\n'
'\n \n1871 Broadway\r\nNew York, NY 10023\n\n212.613.5536\n'

输出:

print ("Venue: " + venue.text + info.text)

我希望它能够为所有不同的场地做到这一点,而不仅仅是一个。我试过循环,但它们似乎只是反复显示一个实例。

1 个答案:

答案 0 :(得分:0)

如果不确切知道display()的作用,我的猜测是你没有打印所有"实例"因为你只是从两个for循环的最后一次迭代中得到结果。

如果没有结构变化,最快的方法是将两个bs4剪贴簿的输出保存到两个列表中,并使用循环显示其内容。

from bs4 import BeautifulSoup
import urllib.request
import sys

def get_HTML(url):
    response = urllib.request.urlopen(url)
    html = response.read()
    return html

venues_html = get_HTML('http://www.cxra.com/venues/new-york/')
soup = BeautifulSoup(venues_html, "lxml")
print('data scrapped')

list1 = []
list2 = []

for venue in soup('a', attrs={'href' : '#', 'onclick' : 'return false;'}):
     list1.append(venue.text)

for info in soup.findAll('div', attrs={'class' : 'infoUnit col-md-6'}):
    list2.append(info.text)

if len(list1) != len(list2):
    print('Two lists not aligned, aborting')
    sys.exit(0)

for index in range(len(list1)):
    try:
        print ("Venue: ", list1[index],  list2[index])
    except Exception as e:
        print(e)