无法使用beautifulsoup3从json脚本提取所有URL

Question

   import requests
from bs4 import BeautifulSoup
import json
import re

url = "https://www.daraz.pk/catalog/?q=dell&_keyori=ss&from=input&spm=a2a0e.searchlist.search.go.57446b5079XMO8"
page = requests.get(url)

print(page.status_code)
print(page.text)
soup = BeautifulSoup(page.text, 'html.parser')
print(soup.prettify())



alpha = soup.find_all('script',{'type':'application/ld+json'})

jsonObj = json.loads(alpha[1].text)

下面是从json对象中查找所有相关产品信息的代码

for item in jsonObj['itemListElement']:
    name = item['name']
    price = item['offers']['price']
    currency = item['offers']['priceCurrency']
    availability = item['offers']['availability'].split('/')[-1]
    availability = [s for s in re.split("([A-Z][^A-Z]*)", availability) if s]
    availability = ' '.join(availability)

这是为json脚本提取URL的代码

    url = item['url']

print('Availability: %s  Price: %0.2f %s   Name: %s' %(availability,float(price), currency,name, url))

下面是从csv中提取数据的代码：

outfile = open('products.csv','w', newline='')
writer = csv.writer(outfile)
writer.writerow(["name", "type", "price", "priceCurrency", "availability" ])

alpha = soup.find_all('script',{'type':'application/ld+json'})

jsonObj = json.loads(alpha[1].text)

for item in jsonObj['itemListElement']:
    name = item['name']
    type = item['@type']
    url = item['url']
    price = item['offers']['price']
    currency = item['offers']['priceCurrency']
    availability = item['offers']['availability'].split('/')[-1]

文件创建了标头，但没有CSV格式的URL数据

writer.writerow([name, type, price, currency, availability, URL ])
outfile.close()

Answer 1

首先，您不在其中包含标题。没什么大不了的，只是第一行的url列中的标题为空白。因此，包括：

writer.writerow(["name", "type", "price", "priceCurrency", "availability", "url" ]) 

第二，将字符串存储为url，然后在编写器中引用URL。 URL没有任何价值。实际上，它应该给出URL is not defined或类似的错误。

并且由于您已经在代码中使用url使用了url = "https://www.daraz.pk/catalog/?q=dell&_keyori=ss&from=input&spm=a2a0e.searchlist.search.go.57446b5079XMO8"，所以我也可能会将变量名更改为url_text。

我可能还会使用变量type_text或type以外的其他变量，因为type是python中的内置函数。

但是您需要更改为：

writer.writerow([name, type, price, currency, availability, url ])
outfile.close()

完整代码：

import requests
from bs4 import BeautifulSoup
import json
import csv

url = "https://www.daraz.pk/catalog/?q=dell&_keyori=ss&from=input&spm=a2a0e.searchlist.search.go.57446b5079XMO8"
page = requests.get(url)

print(page.status_code)
print(page.text)
soup = BeautifulSoup(page.text, 'html.parser')
print(soup.prettify())



alpha = soup.find_all('script',{'type':'application/ld+json'})

jsonObj = json.loads(alpha[1].text)

outfile = open('c:\products.csv','w', newline='')
writer = csv.writer(outfile)
writer.writerow(["name", "type", "price", "priceCurrency", "availability" , "url"])

for item in jsonObj['itemListElement']:
    name = item['name']
    type_text = item['@type']
    url_text = item['url']
    price = item['offers']['price']
    currency = item['offers']['priceCurrency']
    availability = item['offers']['availability'].split('/')[-1]

    writer.writerow([name, type_text, price, currency, availability, url_text ])

outfile.close()

Answer 2

我唯一发现错误的地方是，您在最后一行输入错字-大写URL而不是小写url。对其进行更改可以使脚本完美运行。