我在Python中使用scrapy，想要检索另一个元素的内容，然后再扩展＆＃34;元件。在检查DOM树时，在首次单击父元素之前，不会加载div标记和文本本身。单击父级后，可以重新隐藏文本，但至少会在DOM中。
示例网站是here。我正在寻找抽象文本（在单击＆＃34; Abstract＆＃34;链接之前未加载）。
scrappy命令是：
response.xpath("//div[@class='previewBox abstract hidden']").extract()但是会返回一堆像这样的空div：
u'<div id="abs_S0740002015000179" class="previewBox abstract hidden"></div>'
如果我使用它：response.xpath("//div[@class='previewBox abstract']").extract()那么它根本不会返回任何内容。
答案 0 :(得分：1)
您需要模拟在abtract链接点击时发送的其他HTTP GET请求。
我们的想法是提取并请求“摘要”链接的data-url属性值。
$ scrapy shell "http://www.sciencedirect.com/science?_ob=ArticleListURL&_method=list&_ArticleListID=-764831607&_sort=r&_st=13&view=c&md5=a41e9f25739feae932862575251c1e0d&searchtype=a"
In [1]: url = response.xpath("//a[@data-type='abstract']/@data-url").extract()[0]
In [2]: fetch(url)
In [3]: print "".join(response.xpath("//div[@class='articleText']//text()").extract())
AbstractThe aim of the present study was to investigate the effect of lactic acid against Shiga toxin producing Escherichia coli (O157:H7 and non-O157 serogroups including O103, O111, O145 and O26) at different conditions. Soybean sprouts and spinach leaves inoculated with each serogroup of E. coli (∼7.00 + 1.00 log10 cfu/g) were treated with the lactic acid solutions at different concentrations (0% (control), 1.5%, 2.0%, or 2.5%) and at different temperatures (20, 40, or 50 °C) for 3 min. Results indicated that regardless of the treatment temperature, no significant reduction in the numbers of any serogroup occurred in the control group (0%) (p > 0.05). However, lactic acid at concentration of 1.5%, 2% and 2.5% was found to be effective against all organisms tested. There was no significant difference (p > 0.05) between E. coli O157:H7 and non-O157 STEC serogroups at any treatment group. The highest reductions (ca. 4.00 log10 cfu/g) of all serotypes in both produces were observed after immersing into 2.5% lactic acid at 50 °C. The results of this study showed that decontamination of fresh produces such as spinach and soybean sprout with lactic acid solutions prepared at mild temperatures (40 °C and 50 °C) might be an effective safety measure in preventing public health risks associated with these products contaminated with STEC.
请注意，此fetch()调用是在shell中发出其他请求的特殊方法。在Scrapy蜘蛛中，您需要yield或return scrapy.http.Request()实例，并在callback中解析结果。