Question

我在使用美丽汤进行Web抓取时遇到问题，这是URL http://desiopt.com/company/4316/VST-CONSULTING-INC/，我正尝试通过Web抓取公司信息详细信息。

from selenium import webdriver
import bs4
import pandas as pd
from bs4 import BeautifulSoup
import re
driver =  webdriver.Chrome(executable_path=r"C:/Users/Chandra Sekhar/Desktop/chrome-driver/chromedriver.exe")
titles=[]
driver.get("http://desiopt.com/company/4316/VST-CONSULTING-INC/")
content = driver.page_source
soup = BeautifulSoup(content)
for a in soup.findAll('div',href=True, attrs={'class':'headerBgBlock'}):
    title=a.find('div', attrs={'class':'userInfo'})
    print(title.text)
    df = pd.DataFrame({'Product Title':titles})
    df['Price'] = df['Price'].map(lambda x: re.sub(r'\W+', '', x))
    df.to_csv('products1.csv', index=False)

Answer 1

import requests
from bs4 import BeautifulSoup

r = requests.get('http://desiopt.com/company/4316/VST-CONSULTING-INC/')
soup = BeautifulSoup(r.text, 'html.parser')


for item in soup.findAll('div', attrs={'class': 'compProfileInfo'}):
    for a in item.findAll('span'):
          print(a.text.strip())

输出：

VST CONSULTING INC
Phone
732-491-8681
Email
bindu@vstconsulting.com
Web Site
www.vstconsulting.com

关于网页抓取的问题

1 个答案: