Question

我正在尝试从alibaba.com提取一些数据。为此，我正在使用scrapy。尽管它适用于大多数零件，但选择器似乎并未从公司资料中获取代码块。谁能帮助我解决这个问题？

# -*- coding: utf-8 -*-
import scrapy
import csv
import os
import numpy as np

class AlibabaCrawlerSpider(scrapy.Spider):
    name = 'alibaba_crawler'
    allowed_domains = ['alibaba.com']
    start_urls = ['http://alibaba.com/']
    delimiter = '|'

    def start_requests(self):
        """Read keywords from keywords file amd construct the search URL"""

        with open(os.path.join(os.path.dirname(__file__), "../resources/keywords.csv")) as search_keywords:
            for keyword in csv.DictReader(search_keywords):
                search_text=keyword["keyword"]
                url="https://www.alibaba.com/trade/search?fsb=y&IndexArea=product_en&CatId=&SearchText={0}&viewtype=G".format(
                    search_text)
                # The meta is used to send our search text into the parser as metadata
                yield scrapy.Request(url, callback = self.parse, meta = {"search_text": search_text})


    def parse(self, response):
        """Function to process alibaba search results page"""
        search_keyword=response.meta["search_text"]
        products=response.xpath("//div[@class='item-main']")

        # Defining the XPaths

        XPATH_PRODUCT_LINK=".//div[@class='item-info']//h2/a/@href"

        # iterating over search results
        for product in products:

            raw_product_link=product.xpath(XPATH_PRODUCT_LINK).extract()

            print(raw_product_link)

            product_link="https:" + raw_product_link[0] if raw_product_link else None

            yield scrapy.Request(product_link, callback=self.parse_product)

            break

    def parse_product(self, response):

        product=response.xpath("//div[@class='content-body']")

        # Defining the XPaths

        XPATH_COMPANY_FIELD=".//div[@class='tab-body']//div[contains(@class,'ls-company')]"#//div[@class='alisite']"#td[@class='field-title']/text()"

        raw_company_field=product.xpath(XPATH_COMPANY_FIELD) #.extract()

        print(raw_company_field)

我正在尝试打印raw_company_field。到目前为止，它一直有效。但是当我进入低于等等。enter image description here

Answer 1

XPath不会以这种方式检查类。

像//div[@class='tab-body']这样的选择器将仅以tab-body作为其匹配类。要选择其他具有类的元素，可以执行以下操作：

//div[contains(concat(' ',normalize-space(@class),' '),' tab-body ')]

或改用CSS选择器：

div.tag-body

XPATH选择器无法选择HTML代码块

1 个答案: