Question

我已经用python创建了一个脚本，该脚本能够从网页收集数据并将其存储到M = zeros(NewX, NewY,class(Igray));中。将数据正确插入mysql后，我的脚本可以在控制台中打印它们。

我的问题是：如何将以下三行包装在一个单独的函数中并打印存储中的数据？

mysql

我的完整剧本：

mycursor.execute("SELECT * FROM webdata")
for item in mycursor.fetchall():
    print(item)

Answer 1

我的评论旨在描述此更改的代码：

import mysql.connector
from bs4 import BeautifulSoup
import requests

# This is a new function
def do_fetch (cursor):
    cursor.execute("SELECT * FROM webdata")
    for item in cursor.fetchall():
        print(item)

URL = "https://www.tripadvisor.com.au/Restaurants-g255068-c8-Brisbane_Brisbane_Region_Queensland.html"

def get_info(link):
    mydb = mysql.connector.connect(
      host="localhost",
      user="root",
      passwd = "123",
      database="mydatabase"
    )
    mycursor = mydb.cursor()
    mycursor.execute("DROP TABLE if exists webdata")
    mycursor.execute("CREATE TABLE if not exists webdata (name VARCHAR(255), bubble VARCHAR(255), review VARCHAR(255))")

    response = requests.get(link)
    soup = BeautifulSoup(response.text,"lxml")
    for items in soup.find_all(class_="shortSellDetails"):
        name = items.find(class_="property_title").get_text(strip=True)
        bubble = items.find(class_="ui_bubble_rating").get("alt")
        review = items.find(class_="reviewCount").get_text(strip=True)

        mycursor.execute("INSERT INTO webdata (name,bubble,review) VALUES (%s,%s,%s)",(name,bubble,review))
        mydb.commit()

    # This part is changed (moved to new function):
    do_fetch (mycursor)

if __name__ == '__main__':
    get_info(URL)

Answer 2

我认为以下是如何将事物分成不同的功能并根据需要打印数据。确保像建议 here 一样在for循环之外提交。

import mysql.connector
from bs4 import BeautifulSoup
import requests

url = "https://www.tripadvisor.com.au/Restaurants-g255068-c8-Brisbane_Brisbane_Region_Queensland.html"

def connect():
    mydb = mysql.connector.connect(
      host="localhost",
      user="root",
      passwd = "123",
      database="mydatabase"
    )
    return mydb

def create_table(link):
    conn = connect()
    mycursor = conn.cursor()
    mycursor.execute("DROP TABLE if exists webdata")
    mycursor.execute("CREATE TABLE if not exists webdata (name VARCHAR(255), bubble VARCHAR(255), review VARCHAR(255))")

    response = requests.get(link)
    soup = BeautifulSoup(response.text,"lxml")
    for items in soup.find_all(class_="shortSellDetails"):
        name = items.find(class_="property_title").get_text(strip=True)
        bubble = items.find(class_="ui_bubble_rating").get("alt")
        review = items.find(class_="reviewCount").get_text(strip=True)
        mycursor.execute("INSERT INTO webdata (name,bubble,review) VALUES (%s,%s,%s)",(name,bubble,review))
    conn.commit() #this is where you should commit

def fetch_data():
    conn = connect()
    mycursor = conn.cursor()
    mycursor.execute("SELECT * FROM webdata")
    for item in mycursor.fetchall():
        print(item)

if __name__ == '__main__':
    create_table(url)
    fetch_data()

Answer 3

您是否考虑过使用Django模型框架，它允许您以python对象的形式与任何表进行交互。

在现有数据库中，您可以按照here的说明将其转换为模型，这只需在Django设置文件中设置 db 即可。

DATABASES = {
'default': {
    'ENGINE': 'django.db.backends.mysql', 
    'NAME': 'DB_NAME',
    'USER': 'DB_USER',
    'PASSWORD': 'DB_PASSWORD',
    'HOST': 'localhost',   # Or an IP Address that your DB is hosted on
    'PORT': '3306',
    }
}

然后您可以使用django ops从表中创建模型

$ python manage.py inspectdb > models.py

然后您将在models.py文件中得到一些对象定义，然后可以访问它们。

您将最终得到一个这样的对象（对数据库没有太多了解），您只需要向其添加一个__str__()句柄，以便在打印该对象时正确设置其格式：

from django.db import models

class ShortSellDetails(models.Model):
    id = models.UUIDField(default=uuid.uuid4, editable=False, unique=True)
    name = models.CharField()
    bubble = models.CharField()
    review = models.TextField()

    def __str__(self):
        return "VALUES ({0},{1}, {2})".format(
             self.name,self.bubble,self.review)

然后您就可以像所有models一样与该对象进行交互（确保将模型添加到INSTALLED APPS中，例如shortSell并执行./manage.py makemigrations & ./manage.py migrate），例如：

from shortSell import ShortSellDetails

details = ShortSellDetails.object.all()
for detail in details:
    print(detail)

或者您也可以直接使用模型界面直接创建对象

from bs4 import BeautifulSoup
import requests
from shortSell import ShortSellDetails

URL = "https://www.tripadvisor.com.au/Restaurants-g255068-c8-Brisbane_Brisbane_Region_Queensland.html"
response = requests.get(URL)
soup = BeautifulSoup(response.text,"lxml")

for items in soup.find_all(class_="shortSellDetails"):
        name = items.find(class_="property_title").get_text(strip=True)
        bubble = items.find(class_="ui_bubble_rating").get("alt")
        review = items.find(class_="reviewCount").get_text(strip=True)


        ShortSellDetails.objects.create(
            name=name, 
            bubble=bubble, 
            review=review)

details = ShortSellDetails.object.all()
for detail in details:
    print(detail)

找不到使用功能从mysql打印数据的任何想法

3 个答案: