通过redis进行后台处理的我的Django开发网站在本地工作,但不适用于heroku。我的requirements.txt包含'redis == 2.7.2'和'rq == 0.3.2',我在heroku上调整了web = 1和worker = 1。我有'redistogo:nano'插件。验证它在heroku控制台上运行的说明不起作用:没有'redis.set()'或'redis.get()'。所以我尝试了Running Heroku background tasks with only 1 web dyno and 0 worker dynos中的代码,如下所示。 在settings.py中:
import os
import os.path
import socket
import sys
import urlparse
from rq import Queue
from worker import conn
redis_q = Queue(connection=conn)
...
这是worker.py:
import os
import redis
from rq import Worker, Queue, Connection
listen = ['high', 'default', 'low']
redis_url = os.getenv('REDISTOGO_URL', 'redis://localhost:6379')
conn = redis.from_url(redis_url)
if __name__ == '__main__':
with Connection(conn):
worker = Worker(map(Queue, listen))
worker.work()
以下是进行后台调用的视图片段:
from upload import * # where process_upload_file() lives
...
catchall = {...}
catchall_string = str(catchall)
redis_q.enqueue(process_upload_file, catchall_string)
...
upload.py有:
def process_upload_file(catchall_string):
...
为什么这不起作用?
(为了额外的功劳,如果有人能给我一个更好的方法将参数作为字符串传递,我们将不胜感激。)
Procfile:
web: gunicorn exim.wsgi -b "0.0.0.0:$PORT"
worker: python worker.py
以下是设置S3存储桶并将heroku指向它后的最新尝试的heroku日志。它无法找到“http://s3.amazonaws.com/mycompany/sample.csv”(此处显示)或“http://mycompany.s3.amazonaws.com/sample.csv”。
0:46:31+00:00 app[worker.1]: [2013-02-09 00:46] INFO: worker:
2013-02-09T00:46:44+00:00 heroku[web.1]: Starting process with command `gunicorn myapp.wsgi -b "0.0.0.0:27999"`
2013-02-09T00:46:45+00:00 app[web.1]: 2013-02-09 00:46:45 [2] [INFO] Starting gunicorn 0.15.0
2013-02-09T00:46:45+00:00 app[web.1]: 2013-02-09 00:46:45 [2] [INFO] Listening at: http://0.0.0.0:27999 (2)
2013-02-09T00:46:45+00:00 app[web.1]: 2013-02-09 00:46:45 [2] [INFO] Using worker: sync
2013-02-09T00:46:45+00:00 app[web.1]: 2013-02-09 00:46:45 [5] [INFO] Booting worker with pid: 5
2013-02-09T00:46:50+00:00 heroku[web.1]: State changed from starting to up
2013-02-09T00:47:15+00:00 heroku[router]: at=info method=GET path=/favicon.ico host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=3ms service=3132ms status=404 bytes=30
2013-02-09T00:47:15+00:00 heroku[router]: at=info method=GET path=/upl/upload75/ host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=6ms service=980ms status=200 bytes=958
2013-02-09T00:47:15+00:00 heroku[router]: at=info method=GET path=/favicon.ico host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=2ms service=16ms status=404 bytes=30
2013-02-09T00:47:21+00:00 heroku[router]: at=info method=POST path=/upl/upload75/ host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=4ms service=244ms status=302 bytes=5
2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/upl/upload77/ host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=2ms service=162ms status=200 bytes=44024
2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/css/bootstrap.min.css host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=1ms connect=3ms service=26ms status=200 bytes=95438
2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/css/style.css host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=2ms service=15ms status=200 bytes=515
2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/js/jquery-1.7.2.min.js host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=3ms service=17ms status=200 bytes=94840
2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/js/scripts.js host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=1ms connect=6ms service=5ms status=200 bytes=509
2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/js/bootstrap.min.js host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=2ms service=7ms status=200 bytes=22323
2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/snippets/footer.html host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=2ms service=16ms status=200 bytes=537
2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/img/groovepaper.png host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=1ms service=11ms status=200 bytes=74596
2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/font/fontawesome-webfont.woff host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=3ms service=12ms status=200 bytes=41752
2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/favicon.ico host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=1ms service=15ms status=404 bytes=30
2013-02-09T00:47:37+00:00 heroku[router]: at=info method=POST path=/upl/upload77/ host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=2ms service=182ms status=200 bytes=98
2013-02-09T00:47:37+00:00 app[worker.1]: [2013-02-09 00:47] INFO: worker: default: upload.process_upload_file("{'server_file': u'/app/myapp/myapp/uploads/d27/org20130208194721.csv', 'client_file': u'aorg.csv', 'myco_id': 27, 'user_email': u'me.myself@mycompany.com', 'ignore_header_line': True, 'content_type': 'org', 'column_choices': {u'org_name': 0, u'addr1': 1, u'country_code': 8}}") (eb79ed7e-cff8-4071-a0c8-b1d4b1bd834f)
2013-02-09T00:47:37+00:00 heroku[router]: at=info method=GET path=/favicon.ico host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=1ms service=9ms status=404 bytes=30
2013-02-09T00:47:42+00:00 app[worker.1]: [2013-02-09 00:47] ERROR: horse: IOError: [Errno 2] No such file or directory: 'http://s3.amazonaws.com/mycompany/sample.csv'
2013-02-09T00:47:42+00:00 app[worker.1]: rv = job.perform()
2013-02-09T00:47:42+00:00 app[worker.1]: File "/app/refactor.py", line 72, in read_uploaded_file
2013-02-09T00:47:42+00:00 app[worker.1]: allrows = csv.reader(open(filepath, 'rU'), dialect='excel')
2013-02-09T00:47:42+00:00 app[worker.1]: IOError: [Errno 2] No such file or directory: 'http://s3.amazonaws.com/mycompany/sample.csv'
2013-02-09T00:47:42+00:00 app[worker.1]: File "/app/refactor.py", line 41, in read_csv
2013-02-09T00:47:42+00:00 app[worker.1]: [2013-02-09 00:47] WARNING: horse: Moving job to failed queue.
2013-02-09T00:47:42+00:00 app[worker.1]: Traceback (most recent call last):
2013-02-09T00:47:42+00:00 app[worker.1]: output, nrows, error = acceptable_upload_filetypes[ext](filepath, **kwargs)
2013-02-09T00:47:42+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/rq/worker.py", line 393, in perform_job
2013-02-09T00:47:42+00:00 app[worker.1]:
2013-02-09T00:47:42+00:00 app[worker.1]: [2013-02-09 00:47] DEBUG: horse: Invoking exception handler <bound method Worker.move_to_failed_queue of <rq.worker.Worker object at 0x12cb310>>
2013-02-09T00:47:42+00:00 app[worker.1]: File "/app/.heroku/python/lib/python2.7/site-packages/rq/job.py", line 328, in perform
2013-02-09T00:47:42+00:00 app[worker.1]: self._result = self.func(*self.args, **self.kwargs)
2013-02-09T00:47:42+00:00 app[worker.1]: File "/app/upload.py", line 248, in process_upload_file
2013-02-09T00:47:42+00:00 app[worker.1]: all_rows, nrows, error = read_uploaded_file(server_file, maxrows=maxrows)
2013-02-09T00:47:42+00:00 app[worker.1]: [2013-02-09 00:47] INFO: worker:
2013-02-09T00:47:42+00:00 app[worker.1]: [2013-02-09 00:47] INFO: worker: *** Listening on high, default, low...
(venv)me@host:~/myapp$