Question

因此，我的问题来自网页中的特定标签，该标签隐藏了我要剪贴的内容。

让我们告诉您我确切希望取消的内容。

由于我无法插入图片，因为我错过了它的声誉，因此给了我一些指向图片主机的链接。

您可以在其中看到，“ COTES”标签的内容实际上隐藏在html渲染中，就像您在红框中看到的一样。

但是我注意到，如果我将< div id="pariCotesTab" class="tab" style="display: none;"的属性style更改为style="display: block;"，则会显示隐藏的html部分：you can see it in the second image。

当我使用scrapy shell https://www.zeturf.fr/fr/course/2018-10-19/R1C1-vincennes-prix-klymene/turf并尝试获取“ COTES”选项卡的元素时，例如In [1]: response.xpath("//td[@class='cote-simplegagnant cote-reference']/text()").extract()，它什么也不返回Out 1: []，这是合乎逻辑的，但它阻止了我。 / p>

那么如何在匆忙的过程中将属性style更改为style="display: block;"？并在“ COTES”标签中获取要剪贴的内容。

我想提一提，我宁愿不使用飞溅，因为这对我来说安装docker非常麻烦……我不会使用硒，因为我想剪贴约1000页，也许更多。我在前一个项目中使用了硒，而使用任何浏览器进行网络驱动只是浪费时间。首先，这是一个Web测试程序，而不是WebCrawler和Webscraper模块。

Scrapy版本：1.5.0，Python版本：2.7.9

Answer 1

您的问题不是您想像的那样。

Scrapy不在乎使用哪种CSS样式（除非您在选择器中显式使用它们），而仅关心页面源代码中存在的样式。
页面的来源包含以下内容：

<th class="cote-simplegagnant cote-reference"></th>

如您所见，您选择的th是空的，稍后将由javascript填充。
仔细查看源代码，您可以找到包含所需信息的script标签：

cotesInfos: {"referenceDateTime":{"date":"2018-10-19 19:30:00.000000","timezone_type":3,"timezone":"Europe\/Paris"},"meilleureCoteSG":{"reference":3.3,"live":3.6},"displayedColumns":{"hasSG":1,"hasSP":1,"hasZC":1,"hasZS":1},"1":{"odds_single":27.2,"odds_couillon":11.7,"odds_zeshow":29,"odds":{"reference":36.4,"SG":27.2,"SPMin":5,"SPMax":9.1,"ZC":11.7,"ZS":29},"oddsprogress_single":"moins"},"2":{"odds_single":13.3,"odds_couillon":13.6,"odds_zeshow":11.4,"odds":{"reference":14.5,"SG":13.3,"SPMin":2.3,"SPMax":4.1,"ZC":13.6,"ZS":11.4}},"3":{"odds_single":3.7,"odds_couillon":7.2,"odds_zeshow":8,"odds":{"reference":6.8,"SG":3.7,"SPMin":1.2,"SPMax":1.8,"ZC":7.2,"ZS":8},"oddsprogress_single":"moins"},"4":{"odds_single":274.1,"odds_couillon":19.6,"odds_zeshow":40.9,"odds":{"reference":168.9,"SG":274.1,"SPMin":13.5,"SPMax":41.7,"ZC":19.6,"ZS":40.9},"oddsprogress_single":"plus"},"5":{"odds_single":20.2,"odds_couillon":9,"odds_zeshow":13.1,"odds":{"reference":16,"SG":20.2,"SPMin":2.9,"SPMax":5.2,"ZC":9,"ZS":13.1},"oddsprogress_single":"plus"},"6":{"odds_single":9.4,"odds_couillon":11.7,"odds_zeshow":12.6,"odds":{"reference":4.8,"SG":9.4,"SPMin":3.2,"SPMax":5.8,"ZC":11.7,"ZS":12.6},"oddsprogress_single":"plus"},"7":{"odds_single":32.3,"odds_couillon":9.8,"odds_zeshow":11.4,"odds":{"reference":27.9,"SG":32.3,"SPMin":5.1,"SPMax":9.2,"ZC":9.8,"ZS":11.4},"oddsprogress_single":"plus"},"8":{"odds_single":78.2,"odds_couillon":16.3,"odds_zeshow":34.8,"odds":{"reference":109.3,"SG":78.2,"SPMin":8,"SPMax":14.7,"ZC":16.3,"ZS":34.8},"oddsprogress_single":"moins"},"9":{"odds_single":7.1,"odds_couillon":9.9,"odds_zeshow":9.5,"odds":{"reference":11.2,"SG":7.1,"SPMin":1.5,"SPMax":2.5,"ZC":9.9,"ZS":9.5},"oddsprogress_single":"moins"},"10":{"odds_single":3.6,"odds_couillon":18.9,"odds_zeshow":2.9,"odds":{"reference":3.3,"SG":3.6,"SPMin":1.6,"SPMax":2.7,"ZC":18.9,"ZS":2.9}},"11":{"odds_single":16.4,"odds_couillon":9.6,"odds_zeshow":13.1,"odds":{"reference":14.4,"SG":16.4,"SPMin":3.4,"SPMax":6,"ZC":9.6,"ZS":13.1},"oddsprogress_single":"plus"},"12":{"odds_single":21.3,"odds_couillon":6.7,"odds_zeshow":10,"odds":{"reference":23.3,"SG":21.3,"SPMin":3.8,"SPMax":6.8,"ZC":6.7,"ZS":10}},"13":{"odds_single":40.9,"odds_couillon":21,"odds_zeshow":27.8,"odds":{"reference":20.1,"SG":40.9,"SPMin":5.8,"SPMax":10.6,"ZC":21,"ZS":27.8},"oddsprogress_single":"plus"},"14":{"odds_single":34.8,"odds_couillon":10.8,"odds_zeshow":20.4,"odds":{"reference":22.2,"SG":34.8,"SPMin":5.2,"SPMax":9.5,"ZC":10.8,"ZS":20.4},"oddsprogress_single":"plus"}}

Answer 2

因此，对stranac的回答感到宽慰，我找到了我所需要的。在检查器中，您可以在<script标签的html中看到“ COTES”选项卡。 odds_raw = response.xpath("//script[@type='text/javascript']").extract()[6]惠特我得到了我想要的。但这还很原始。因此，我在教程中使用了Micheal Yin的建议，您可以参考here。这是stack上的另一个示例。

因此，使用命令odds = re.search('({"referenceDateTime")(.*?)(\\n)', odds_raw).group()我得到了

Out[65]: u'{"referenceDateTime":{"date":"2018-10-19 19:30:00.000000","timezone_t
ype":3,"timezone":"Europe\\/Paris"},"meilleureCoteSG":{"reference":3.3,"live":3.
6},"displayedColumns":{"hasSG":1,"hasSP":1,"hasZC":1,"hasZS":1},"1":{"odds_singl
e":27.2,"odds_couillon":11.7,"odds_zeshow":29,"odds":{"reference":36.4,"SG":27.2
,"SPMin":5,"SPMax":9.1,"ZC":11.7,"ZS":29},"oddsprogress_single":"moins"}...}}\n'

现在我使用json.loads函数，json.loads(odds)然后得到了python字典。

如何在抓取过程中通过div更改div标签中样式属性的值？

2 个答案: