Question

我正在使用以下代码抓取网站，然后将数据存储到sqlite表中。我的问题是for n in str(shark):之后的正则表达式，出于某种原因，place, date, article = groups[1], groups[2], groups[3]不存储任何数据，因此无法插入我的数据库中。问题是，当我在REPL group = re.match(r'(.*?)\W+—?\W+On\W+(.*?\d{4})\W*(.*)', str(shark[1]), flags=re.DOTALL)中运行以下代码时，便能够从鲨鱼列表中获取解析出的文本。知道为什么吗？

import pandas as pd

import re ## added 
import bs4
import sqlite3
import requests
import textwrap

'''
Let's pull some fresh shark data!

'''

res = requests.get('http://www.sharkresearchcommittee.com/pacific_coast_shark_news.htm')
res.raise_for_status()
soup = bs4.BeautifulSoup(res.text, 'html.parser')

shark = []
for i in range(1, 100): # attempting to grab the most recent added paragraph 

    elems = soup.select('body > div > div > center > table > tr > td:nth-of-type(2) > p:nth-of-type({})'.format(i))
    for i in elems:
        #print("—" in str(i))
        if '—' in str(i):
            text = bs4.BeautifulSoup(str(i), 'html.parser')
            shark.append(text)
            #print(text)

'''

'''
c = sqlite3.connect('shark.db')
try:
    c.execute('''CREATE TABLE
                    mytable (Location        STRING,
                             Date            STRING,
                             Description     STRING)''')
except sqlite3.OperationalError: #i.e. table exists already
    pass

for n in str(shark):
        groups = re.match(r'(.*?)\W+—?\W+On\W+(.*?\d{4})\W*(.*)', n, flags=re.DOTALL)

        if not groups:
            continue
        place, date, article = groups[1], groups[2], groups[3]
        print(place)

        c.execute('''INSERT INTO mytable(Location, Date, Description) VALUES(?,?,?)''',
            (place, date, article))
c.commit()

'''
Read into python
'''
df = pd.read_sql_query("select * from mytable;",c)

print（df）

Answer 1

问题是str()

for n in str(shark):

它将列表shark转换为单个字符串，但必须分别转换每个元素n

for n in shark:
    n = str(n)

使用BS4进行抓取-使用正则表达式-存储在sqlite中

1 个答案: