Question

我有这个python'2.6.4'脚本：

#!/usr/bin/python
import MySQLdb
import csv
import pprint

db = MySQLdb.connect(host="localhost", # The Host
                     user="username", # username
                      passwd="password", # password
                      db="dbname") # name of the data base

cursor = db.cursor() 

cursor.execute("SELECT name, id, city, storeid FROM Products;")

StudentsData = cursor.fetchall()
pprint.pprint(StudentsData)

此查询返回数百万行，执行3 minutes，每天运行20次，每个用户必须等待三分钟才能获取输出。

我在想caching它，但我不知道如何在python中做到这一点，这将是一个好主意，如果是这样，我怎么能这样做，使用obove代码。

如果有更好的方法可以帮助我吗？

Answer 1

是否缓存的决定完全取决于您。过时的数据会成为问题吗？

我能想到的最简单的缓存方案类似于以下使用Python的pickle module：

import MySQLdb
import csv
import pprint
import time

MAX_CACHE_AGE = 60*20  # 20 Minutes
CACHE_FILENAME = 'results.cache'

with open(CACHE_FILENAME, 'r') as cache:
    cached = pickle.load(cache)

if(time.time() > cached['timestamp'] + MAX_CACHE_AGE):
    # Cache too old, run query
    db = MySQLdb.connect(host="localhost", # The Host
                     user="username", # username
                     passwd="password", # password
                     db="dbname") # name of the data base

    cursor = db.cursor() 
    cursor.execute("SELECT name, id, city, storeid FROM Products;")
    StudentsData = cursor.fetchall()

    # Update cache file
    data = {'results': StudentsData, 'timestamp':time.time()}
    with open(CACHE_FILENAME, 'w') as cache:
        pickle.dump(data, cache)

else:
    # Cached data is fresh enough, use that
    StudentsData = cached['results']

pprint.pprint(StudentsData)

您需要手动初始化results.cache文件。

修改的

with语法是一个上下文管理器，并在Python 2.5中引入。

with open(CACHE_FILENAME, 'r') as cache: cached = pickle.load(cache)

可以改写为

cached = open(CACHE_FILENAME, 'r') cached = pickle.load(cache) cached.close()

的 EDIT2 的

经过长时间的聊天讨论，以下工作：

import MySQLdb import csv import pprint import time import pickle MAX_CACHE_AGE = 60*20 # 20 Minutes CACHE_FILENAME = 'results.cache' regen = False try: with open(CACHE_FILENAME, 'r') as cache: cached = pickle.load(cache) if(time.time() > cached['timestamp'] + MAX_CACHE_AGE): print("Cache too old: regenerating cache") regen = True else: print("Cached data is fresh enough: loading results from cache") except IOError: print("Error opening %s: regenerating cache" % CACHE_FILENAME) regen = True if(regen): # Cache too old, run query db = MySQLdb.connect(host="localhost", # The Host user="username", # username passwd="password", # password db="dbname") # name of the data base cursor = db.cursor() cursor.execute("SELECT name, id, city, storeid FROM Products;") StudentsData = cursor.fetchall() cursor.close() # Update cache file data = {'results': StudentsData, 'timestamp':time.time()} with open(CACHE_FILENAME, 'w') as cache: pickle.dump(data, cache) else: # Cached data is fresh enough, use that StudentsData = cached['results'] print StudentsData

Answer 2

在Python 3中，对于一个非常简单的案例，您可以查看functools.lru_cache：http://docs.python.org/3.2/library/functools.html#functools.lru_cache

对于更复杂的缓存，Beaker可能是最佳选择：http://beaker.readthedocs.org/en/latest/caching.html

缓存Python脚本返回的mysql查询

2 个答案: