Question

我正在使用scrapy构建网络爬虫，我用于检测和检索视频URL的蜘蛛代码为：

all_links = []

class Onlinecrawl(RobotCrowler):

name = "start"
start_urls = ['https://www.w3schools.com/html/html5_video.asp',]

#rules to extract the data, if you want more depth go on website and read article about it : https://doc.scrapy.org/en/0.24/topics/link-extractors.html
rules = (
    Rule(LinkExtractor(

        allow=(['https://www.w3schools.com/html/html5_video.asp']),
        deny=(['/designers', '/edit', '/new-in', '.*\.html']),

        restrict_css=['a'],
        ),
        callback='process_page',
        follow=True,
        ),
)

def parse_page(self, response, starting_url):

    video_details = response.css('video').xpath('@src').extract()
    for video in video_details:
        try:

            all_links.append({"videos": video})
            print (video)


        except (IndexError, TypeError, ValueError) as e:
            self.logger.error(f'Could not add product {product}')
            self.logger.error(str(e))
            # self.increment_dropped_products(starting_url, 1)
            continue

我没有得到id=video1和src mov_bbb.mp4的简单视频。我的目标是在src标签存在的情况下获取video内的所有链接，它们可能处于同一级别或作为子级，并且不确定我的代码在哪里失败。另外，这是我的第一个蜘蛛，因此可以随意对代码进行严格检查，也请注意，该网站不是我要剪贴的唯一网站，我正在使用它进行测试，但是当视频标签存在时，我的代码应该获得所有srcs。这也不是完整的代码__init_subclass__，process_page和get_rules函数，但共享代码应该足够。

我确实去了shell并输入以下内容：

scrapy shell 'https://www.w3schools.com/html/html5_video.asp'

然后：

response.xpath("//@src").extract()

响应：

response.xpath（“ // @ src”）。extract（）

['//static.h-bid.com/w3schools.com/20180525/snhb-w3schools.min.js', '//static.h-bid.com/gdpr/cmp.stub.js', 'mov_bbb.mp4', 'mov_bbb.ogg', 'mov_bbb.mp4', 'mov_bbb.ogg', '/images/colorpicker.gif', '/images/w3schoolscom_gray.gif', '/lib/snigel_w3schools_footer.js', 'https://translate.google.com/translate_a/element.js?cb=googleTranslateElementInit']

我想要的src存在，但是此代码是否使我获得了每个单独的src？我想使用那种方法吗？

即使最后一次外壳测试都失败的示例：

scrapy shell 'http://www.aljazeera.net/portal'

这是一个不同的网站

>>> response.xpath("//@src").extract()
['https://www.google.com/recaptcha/api.js?hl=ar', '//players.brightcove.net/665001584001/BJeiBqmye_default/index.min.js', '/Content/js/newPortalScripts?v=Z-iLKM9YBRO-l1eYY_yx1kcRYdw0lCXmuKIN39AaJC41', 'http://www.geoplugin.net/javascript.gp', 'https://d5nxst8fruw4z.cloudfront.net/atrk.gif?account=R69Tk1ao6C525T', '//www.googleadservices.com/pagead/conversion.js', '//www.googleadservices.com/pagead/conversion/947178488/?label=O6blCOKKl18Q-JfTwwM&guid=ON&script=0', '//nexus.ensighten.com/aljazeera/arabic/Bootstrap.js', '/Content/images/headerlogo.png', '/File/GetImageCustom/dccdaf9d-a073-4d31-a7b5-b39b468c07cd/530/300', '/Views/Shared/shared/images/jazeraLogo 2560 ×1440.png', '/Views/Shared/shared/images/jazeraLogo 2560 ×1440.png', '/Views/Shared/shared/images/jazeraLogo 2560 ×1440.png', '/Views/Shared/shared/images/jazeraLogo 2560 ×1440.png', '/Views/Shared/shared/images/jazeraLogo 2560 ×1440.png', '/Views/Shared/shared/images/jazeraLogo 2560 ×1440.png', '/Views/Shared/shared/images/jazeraLogo 2560 ×1440.png', '/Views/Shared/shared/images/jazeraLogo 2560 ×1440.png', '/Views/Shared/shared/images/jazeraLogo 2560 ×1440.png', 'http://blogs.aljazeera.net//File/GetImageCustom/3f749887-1b94-4bdd-b95c-0b739fd4018f/49/49', 'http://blogs.aljazeera.net//File/GetImageCustom/c0257ffc-bed0-4e95-8abe-e3ca0e55fb61/49/49', 'http://blogs.aljazeera.net//File/GetImageCustom/4eee30f8-6bc2-48d5-b615-970be9ba9149/49/49', 'http://blogs.aljazeera.net//File/GetImageCustom/b14f920f-d249-449b-9c7a-450a860a29bf/49/49', 'http://blogs.aljazeera.net//File/GetImageCustom/eb56d7cc-20f1-4e79-be4e-51af09f4cee0/49/49', 'http://blogs.aljazeera.net//File/GetImageCustom/de0bef7f-4a65-486f-9c3e-802d4617e565/49/49', 'http://blogs.aljazeera.net//File/GetImageCustom/43416467-e38c-4967-ba77-88868c13278d/49/49', '/File/GetImageCustom/00000000-0000-0000-0000-000000000000', '/File/GetImageCustom/f4b80a64-7477-48ce-b280-56fd6ddc76d1/272/158', '/File/GetImageCustom/dccdaf9d-a073-4d31-a7b5-b39b468c07cd/272/158', '/File/GetImageCustom/1b20fec7-ff09-46d9-bd4c-3c3e17bdb640/272/158', '/File/GetImageCustom/52ee7757-ab01-46f1-b192-ea572e0d5ef8/272/158', '/File/GetImageCustom/f254f4e0-f6b2-41c2-9c57-9581dfa57264/272/158', '/File/GetImageCustom/89fe9d07-626b-4479-a195-0c721a240da1/272/158', '/File/GetImageCustom/5196299b-257d-40e0-b76f-b9979427eb6f/272/158', '/File/GetImageCustom/04531520-7ae1-4dec-92fb-6613996bee52/272/158', '/File/GetImageCustom/290f4cb3-8e81-45c2-893f-30208ec476bf/272/158', '/File/GetImageCustom/9bee5bfe-635c-4884-934f-c2d481ecba10/272/158', '/File/GetImageCustom/01c8a89a-2379-4e24-b7b0-a195925324da/272/158', '/File/GetImageCustom/783d5e7b-c34e-4f09-a0fc-1cc903f2af32/272/158', '//platform.twitter.com/widgets.js', '/Content/images/footerlogo.png', '/Content/images/iHorizons-icon.png', '//radar.cedexis.com/1/15249/radar.js']

相同的shell命令，但在此块中无法获取src：

<video id="main-video-9489741_html5_api" data-video-id="5834567349001" data-account="665001584001" data-player="BJeiBqmye" data-embed="default" class="vjs-tech" tabindex="-1" src="blob:http://www.aljazeera.net/9765540b-1e35-4e8c-a304-9e2f9492a07e"></video>

理想情况下，它应该返回blob：http://www.aljazeera.net/9765540b-1e35-4e8c-a304-9e2f9492a07e

编辑：

在评论提及该问题后，失败案例的另一个示例很难复制：

请仔细检查此链接http://www.aljazeera.net/programs/scenarios/2018/9/13/%D8%A7%D9%84%D8%B5%D9%8A%D9%86-%D9%88%D8%A3%D9%81%D8%B1%D9%8A%D9%82%D9%8A%D8%A7-%D8%B4%D8%B1%D8%A7%D9%83%D8%A9-%D9%85%D8%AA%D8%B9%D8%AB%D8%B1%D8%A9-%D8%A3%D9%85-%D9%85%D8%AA%D8%B7%D9%88%D8%B1%D8%A9

并检查其中的视频

您将看到2个类似问题的绝佳示例：

<video id="vid-c8d54140-70fe-4561-9780-8a4e7763c58a_html5_api" data-video-id="5834567349001" data-account="665001584001" data-player="BJeiBqmye" data-embed="default" class="vjs-tech" tabindex="-1" poster="http://bc04.ajmn.me/665001584001/201809/2309/665001584001_5834579342001_5834567349001-vs.jpg?pubId=665001584001" src="blob:http://www.aljazeera.net/a071437f-421f-40ac-9dc7-43afb98d96a0"></video>

再次在此部分中，我要添加blob：http://www.aljazeera.net/a071437f-421f-40ac-9dc7-43afb98d96a0

该集团下的第二个例子：

<div class="vjs-poster" tabindex="-1" aria-disabled="false" style="background-image: url("http://bc04.ajmn.me/665001584001/201809/2309/665001584001_5834579342001_5834567349001-vs.jpg?pubId=66500158;4001");"></div>

我也想从这里http://bc04.ajmn.me/665001584001/201809/2309/665001584001_5834579342001_5834567349001-vs.jpg?pubId=66500158;4001。

Answer 1

加载的视频在很大程度上依赖于javascript，并且似乎为此使用了Brightcove.com 3rd party服务。

如果在加载时打开网络检查器：http://www.aljazeera.net/programs/scenarios/2018/9/13/%D8%A7%D9%84%D8%B5%D9%8A%D9%86-%D9%88%D8%A3%D9%81%D8%B1%D9%8A%D9%82%D9%8A%D8%A7-%D8%B4%D8%B1%D8%A7%D9%83%D8%A9-%D9%85%D8%AA%D8%B9%D8%AB%D8%B1%D8%A9-%D8%A3%D9%85-%D9%85%D8%AA%D8%B7%D9%88%D8%B1%D8%A9

您可以看到正在发出XHR请求，以检索json格式的一些视频数据。它包含图像源URL，例如：

要复制此内容，您需要查看页面html源代码。视频有一个ID：

最后，您可以轻松地将其复制：

from scrapy import Spider, Request
import json

class MySpider(Spider):
    name = 'aljazeera'
    start_urls = ["url with video in it"]
    video_api_url = "https://edge.api.brightcove.com/playback/v1/accounts/665001584001/videos/{id}"

    def parse(self, response):
        video_id = response.css('input#MainVideoDM_BCGuid::attr(value)').extract_first()
        url = self.video_api_url.format(id=video_id)
        yield Request(url, self.parse_video)

    def parse_video(self, response):
        data = json.loads(response.body_as_unicode())
        for source in data['sources']:
            yield source

为什么我的视频搜寻器无法检测到简单的视频src？

1 个答案: