使用python selenium xpath刮脚本标签

Question

比方说，我想从网站上抓取一些元数据：

https://www.diepresse.com/4913597/autocluster-buhlt-um-osterreich-teststrecke-fur-google-autos

更准确地说，即从fullChannel标签的键/home/wirtschaft/international到值<script>：

<script>    
let pageBreakpoint = 'desktop';
let _screen = window.innerWidth;
if (_screen < 640) {
    pageBreakpoint = 'mobile';
} else if (_screen < 1024) {
    pageBreakpoint = 'tablet';
}

var dataLayer = window.dataLayer || [];
    dataLayer.push({
        'siteId': 'dpo',
        'contentId': '4913597',
        'pageType': 'article',
        'contentTitle': 'Autocluster buhlt um Österreich-Teststrecke für Google-Autos',
        'contentAuthor': '',
        'contentElements': '',
        'contentType': 'default',
        'pageTags': '',
        'wordCount': '264',
        'wordCountRounded': '400',
        'contentSource': '',
        'contentPublishingDate': '',
        'contentPublishingDateFormat': '28/01/2016',
        'contentPublishingTime': '08:52',
        'contentPublishingTimestamp': '28/01/2016 08:52:00',
        'contentRepublishingTimestamp': '28/01/2016 08:52:00',
        'contentTemplate': 'default',
        'metaCategory': '',
        'channel': 'international',
        'fullChannel': '/home/wirtschaft/international',
        'canonicalUrl': '',
        'fullUrl': window.location.href,
        'oewaPath': 'RedCont/Wirtschaft/Wirtschaftspolitik',
        'oewaPage': 'homepage',
        'isPremium':'no',
        'isPremiumArticle': 'free',
        'pageBreakpoint': pageBreakpoint,
        'userId': ''
    });
</script>

现在我正在使用Selenium和Xpath，还不能真正弄清楚如何在此上使用正则表达式：

#this doesnt work
driver.find_element_by_xpath("//script[text()]")

有什么建议吗？

Answer 1

您的XPath查找脚本似乎是错误的-尝试以下方法：

script = driver.find_element_by_xpath("//script[contains(text(), 'let pageBreakpoint')]")

然后，您可以使用一些字符串解析方法从fullChannel中提取值：

# Get index of string 'fullChannel'
fullChannelTextIndex = script.index('\'fullChannel\': ')

# Shorten script string by removing everything before 'fullChannel'
simplifiedScript = script[fullChannelTextIndex : len(script)-1]

# Call split on 'canonicalUrl', which appears AFTER 'fullChannel'
# Then replace 'fullChannel' text to get just the field value
fullChannelValue = simplifiedScript .split('\'canonicalUrl\': ')[0].replace('\'fullChannel\': ', '').replace(',', '')

print(fullChannelValue)

这将产生输出'/home/wirtschaft/international'

可能有比通过硒更有效的方法，但是如果您想走这条路线，我将在此处保留硒的答案。

Answer 2

使用JavaScript执行器获取var值datalayer。它将作为json数组返回。

然后获取键fullChannel的值。

driver.get("https://www.diepresse.com/4913597/autocluster-buhlt-um-osterreich-teststrecke-fur-google-autos")
datalayer=driver.execute_script("return dataLayer")
print(datalayer)
print(datalayer[0]['fullChannel'])

输出：

[{'oewaPage': 'homepage', 'contentTitle': 'Autocluster buhlt um Österreich-Teststrecke für Google-Autos', 'userId': '', 'wordCount': '264', 'contentSource': '', 'contentPublishingDate': '', 'contentElements': '', 'contentAuthor': '', 'fullUrl': 'https://www.diepresse.com/4913597/autocluster-buhlt-um-osterreich-teststrecke-fur-google-autos', 'wordCountRounded': '400', 'contentTemplate': 'default', 'canonicalUrl': '', 'contentPublishingTime': '08:52', 'metaCategory': '', 'siteId': 'dpo', 'contentPublishingDateFormat': '28/01/2016', 'isPremium': 'no', 'oewaPath': 'RedCont/Wirtschaft/Wirtschaftspolitik', 'contentRepublishingTimestamp': '28/01/2016 08:52:00', 'contentPublishingTimestamp': '28/01/2016 08:52:00', 'pageTags': '', 'pageBreakpoint': 'desktop', 'contentType': 'default', 'fullChannel': '/home/wirtschaft/international', 'isPremiumArticle': 'free', 'contentId': '4913597', 'channel': 'international', 'pageType': 'article'}, {'faktorVendorData4': 'notset', 'event': 'faktorData', 'faktorData4': 'notset', 'gtm.uniqueEventId': 9, 'faktorData1': 'notset', 'faktorData2': 'notset', 'faktorData5': 'notset', 'faktorData3': 'notset'}, {'gtm.uniqueEventId': 3, 'gtm.start': 1569877670044, 'event': 'gtm.js'}, {'aboStatus': '', 'userId': '', 'userType': 'default', 'userStatus': 'logout'}, {'gtm.uniqueEventId': 6, 'event': 'gtm.dom'}, {'gtm.uniqueEventId': 14, 'gtm.start': 1569877672926, 'event': 'gtm.js'}, {'faktorGdprApplies': 1}, {'gtm.uniqueEventId': 15, 'event': 'gtm.load'}]

键值fullChannel

/home/wirtschaft/international