Question

嘿，所以我有一些抓取html的经验，但是从来没有json，因此我需要使用scrapy http://www.starcitygames.com/buylist/search?search-type=category&id=5061抓取以下网页，并且我在网上找到了一个教程，该教程使用scrapy和jmspath来抓取json数据网络。而且我使该教程正常工作，但是我试图对其进行更改以使其与我的网站兼容，但是没有运气。没有错误，但它不返回任何数据。任何帮助将不胜感激！

items.py

import scrapy


class NameItem(scrapy.Item):
    """User item definition for jsonplaceholder /LoginSpider endpoint."""
    name = scrapy.Field()
    condition = scrapy.Field()
    price = scrapy.Field()
    rarity = scrapy.Field()

LoginSpider.py

import scrapy
import json
from scrapy.spiders import Spider
from scrapy_splash import SplashRequest
from ..items import NameItem
from scrapy.loader import ItemLoader
from scrapy.loader.processors import Join, MapCompose, SelectJmes


class UserSpider(scrapy.Spider):
    """Spider to scrape `http://www.starcitygames.com/buylist/search?search-type=category&id=5061`."""
    name = 'LoginSpider'
    allowed_domains = ['http://www.starcitygames.com/buylist/search?search-type=category&id=5061']
    start_urls = ['http://www.starcitygames.com/buylist/search?search-type=category&id=5061']
    # dictionary to map UserItem fields to Jmes query paths
    jmes_paths = {
            'name': 'name',
            'condition': 'condition',
            'price': 'price',
            'rarity': 'rarity',
            }

    def parse(self, response):
        jsonresponse = json.loads(response.body_as_unicode())
        for user in jsonresponse:
            loader = ItemLoader(item=NameItem())  # create an ItemLoader to populate a NameItem
            loader.default_input_processor = MapCompose(str)  # apply str conversion on each value
            loader.default_output_processor = Join(' ')
            for (field, path) in self.jmes_paths.items():
                loader.add_value(field, SelectJmes(path)(user))
            yield loader.load_item()

Answer 1

此网址http://www.starcitygames.com/buylist/search?search-type=category&id=5061has的响应分为3个级别：

“好”
“搜索”
“结果” ##包含数据

和结果键具有多个您应该迭代的值。值内部是数据。尝试使用此代码，希望对您有所帮助。

这是模块items.py

class SoResponseItem(scrapy.Item):
        name = scrapy.Field()
        condition = scrapy.Field()
        price = scrapy.Field()
        rarity = scrapy.Field()

这是蜘蛛

import scrapy
import json
from SO_response.items import SoResponseItem

class LoginspiderSpider(scrapy.Spider):
    name = 'LoginSpider'
    allowed_domains = ['www.starcitygames.com']
    url = 'http://www.starcitygames.com/'

    def start_requests(self):
        yield scrapy.Request(url=self.url, callback=self.parse)

    def parse(self, response):
        url = response.urljoin('buylist/search?search-type=category&id=5061')
        yield scrapy.Request(url=url, callback=self.parse_data)

    def parse_data(self, response):
        jsonreponse = json.loads(response.body)
        for result in jsonreponse['results']:
            for index in range(len(result)):
                items = SoResponseItem()
                items['name'] = result[index]['name']
                items['condition'] = result[index]['condition']
                items['price'] = result[index]['price']
                items['rarity'] = result[index]['rarity']
                yield items

在您的shell中尝试：抓取抓取-o jmes.json

如何抓取JSON网页

1 个答案: