Question

作为一个有趣的辅助项目，我一直在尝试解析网站以获取当日的随机事实。我决定今天尝试使用BeautifulSoup4和urllib3来解决这个问题。可悲的是，但是我不确定如何比我已经掌握的更深入。

这是我当前的输出水平：

{
"@context": "http://schema.org",
"@type": "Article",
"headline": "Fact of the Day: 51 Facts Of the Day for 3/19/2019 ←FACTSlides→",
"image": "https://www.FACTSlides.com/imgs/ishots/8224.png",
"author": "Luke Reiner",
"genre": "facts",
"publisher": {
    "@type": "Organization",
    "name": "FACTSlides",
    "logo": {
        "@type": "ImageObject",
        "url": "https:\/\/www.factslides.com\/imgs\/logo.png"
    }
},
"url": "https://www.factslides.com/s-Fact-Of-The-Day",
"mainEntityOfPage": "https://www.factslides.com/s-Fact-Of-The-Day",
"datePublished": "2019-03-19",
"dateCreated": "2019-03-19",
"dateModified": "2019-03-19",
"description": "Description.",
"articleBody": "Article clutter here."
}

事实本身存储在articleBody下，并且没有定界，我打算使用'。 “如果我走了那么远，就作为定界符。

这是我到目前为止的代码：

""" Get a random fact. """
import argparse
import json
import urllib3
from bs4 import BeautifulSoup

PARAMETERS = {
    "u": ["url", "passes in a url.", "1"],
}

PARSER = argparse.ArgumentParser(
    description="Arguments to parse a url."
)

HTTP = urllib3.PoolManager()


def __load_args(parser, cfg_list):
    """ Loads the passed arguments. """
    for cfg_key in cfg_list:
        if len(cfg_list[cfg_key]) > 3:
            parser.add_argument(
                "-" + cfg_key,
                "--" + cfg_list[cfg_key][0],
                help=cfg_list[cfg_key][1],
                action=cfg_list[cfg_key][2],
                nargs=cfg_list[cfg_key][3],
            )
        else:
            parser.add_argument(
                "-" + cfg_key,
                "--" + cfg_list[cfg_key][0],
                default=None,
                help=cfg_list[cfg_key][1],
            )


def parse_args(parser, section_list=[]):
    """ Parses the loaded arguments. """
    for section in section_list:
        __load_args(parser, section)
    return parser.parse_args()


ARGS = parse_args(PARSER, [PARAMETERS])

RESPONSE = HTTP.request('GET', ARGS.url)

SOUP = BeautifulSoup(RESPONSE.data, features="html.parser")

SOUP_SCRIPT = SOUP.find_all("script")

JS_TEXT = SOUP.find('script', type='application/ld+json').text

print(JS_TEXT)

任何帮助将不胜感激。

注意：我为事实而解析的网址是here。

Answer 1

只要您的json是文本/字符串，就可以使用json.loads()来读取它：

import json

JS_TEXT = '''{
"@context": "http://schema.org",
"@type": "Article",
"headline": "Fact of the Day: 51 Facts Of the Day for 3/19/2019 ←FACTSlides→",
"image": "https://www.FACTSlides.com/imgs/ishots/8224.png",
"author": "Luke Reiner",
"genre": "facts",
"publisher": {
    "@type": "Organization",
    "name": "FACTSlides",
    "logo": {
        "@type": "ImageObject",
        "url": "https:\/\/www.factslides.com\/imgs\/logo.png"
    }
},
"url": "https://www.factslides.com/s-Fact-Of-The-Day",
"mainEntityOfPage": "https://www.factslides.com/s-Fact-Of-The-Day",
"datePublished": "2019-03-19",
"dateCreated": "2019-03-19",
"dateModified": "2019-03-19",
"description": "Description.",
"articleBody": "Article clutter here."
}'''


jsonObj = json.loads(JS_TEXT)

Answer 2

我在@chitown88的帮助下解决了这个问题。

这里是起作用的代码：

""" Get a random fact. """
import argparse
import random
import json
from bs4 import BeautifulSoup
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

PARAMETERS = {
    "u": ["url", "passes in a url.", "1"],
}

PARSER = argparse.ArgumentParser(
    description="Arguments to parse a url."
)

HTTP = urllib3.PoolManager()


def __load_args(parser, cfg_list):
    """ Loads the passed arguments. """
    for cfg_key in cfg_list:
        if len(cfg_list[cfg_key]) > 3:
            parser.add_argument(
                "-" + cfg_key,
                "--" + cfg_list[cfg_key][0],
                help=cfg_list[cfg_key][1],
                action=cfg_list[cfg_key][2],
                nargs=cfg_list[cfg_key][3],
            )
        else:
            parser.add_argument(
                "-" + cfg_key,
                "--" + cfg_list[cfg_key][0],
                default=None,
                help=cfg_list[cfg_key][1],
            )


def parse_args(parser, section_list=[]):
    """ Parses the loaded arguments. """
    for section in section_list:
        __load_args(parser, section)
    return parser.parse_args()


ARGS = parse_args(PARSER, [PARAMETERS])

RESPONSE = HTTP.request('GET', ARGS.url)

SOUP = BeautifulSoup(RESPONSE.data, features="html.parser")

SOUP_SCRIPT = SOUP.find_all("script")

JS_TEXT = SOUP.find('script', type='application/ld+json').text

JSON_OBJ = json.loads(JS_TEXT)

LIST_TEST = []
# print(JSON_OBJ['articleBody'])
for item in JSON_OBJ['articleBody'].split('. '):
    LIST_TEST.append(item.strip())

print(random.choice(LIST_TEST) + ".")

我想指出，我的范围不是最好的，因为某些“事实”分布在两个句子中。

BeautifulSoup-试图解析网站，但不确定如何解析json脚本？

2 个答案: