无法使用报纸3k下载文章

时间:2018-08-10 21:00:53

标签: python web-scraping python-newspaper

我什至尝试了pypi.org中的命令,但没有下载任何文章。

from newspaper import Article

url = 'http://fox13now.com/2013/12/30/new-year-new-laws-obamacare-pot-guns-and-drones/'
article = Article(url)
article.download()
article.html

article.html仅提供空字符串''。当我尝试article.parse()时出现错误*

  

您必须先download()篇文章!

我尝试了解决方法

while article.download_state == ArticleDownloadState.NOT_STARTED:
    # Raise exception if article download state does not change after 10 seconds
    if slept > 9:
        raise ArticleException('Download never started')
    sleep(1)
    slept += 1

仍然无法解决问题。

1 个答案:

答案 0 :(得分:0)

有时您需要清理链接,例如从RSS供稿。

urlparse python库可用于Google快讯。

示例

google_url = 'https://www.google.com/url?rct=j&sa=t&url=https://www.timesnownews.com/international/article/european-union-chief-donald-tusk-lashes-out-at-donald-trump-stance-on-europe/311933&ct=ga&cd=CAIyHDlhZGYyMmM4NzAwYzNlZDc6Y28udWs6ZW46R0I&usg=AFQjCNHrsEaxxjXvWB3wM_1aRjNg6aeZvw'

在url =之后获取变量

from urllib.parse import urlparse, parse_qs
url = urlparse(google_url)
print (parse_qs(url.query)['url'][0])

此外,还请注意,如果未单独分配不同的输出,则会覆盖输出。

在测试脚本期间,输出仅包含article.text:

article = Article('https://www.google.com/url?rct=j&sa=t&url=https://www.timesnownews.com/international/article/european-union-chief-donald-tusk-lashes-out-at-donald-trump-stance-on-europe/311933&ct=ga&cd=CAIyHDlhZGYyMmM4NzAwYzNlZDc6Y28udWs6ZW46R0I&usg=AFQjCNHrsEaxxjXvWB3wM_1aRjNg6aeZvw')
article.download()
article.parse()
article.top_image
article.text

这在测试脚本期间有效:

top_image = article.top_image
text = article.text
print (top_image, text)