如何让redistogo在heroku上工作?

时间:2013-02-05 18:57:16

标签: django heroku python-2.7 redis redistogo

通过redis进行后台处理的我的Django开发网站在本地工作,但不适用于heroku。我的requirements.txt包含'redis == 2.7.2'和'rq == 0.3.2',我在heroku上调整了web = 1和worker = 1。我有'redistogo:nano'插件。验证它在heroku控制台上运行的说明不起作用:没有'redis.set()'或'redis.get()'。所以我尝试了Running Heroku background tasks with only 1 web dyno and 0 worker dynos中的代码,如下所示。 在settings.py中:

import os
import os.path
import socket
import sys
import urlparse
from rq import Queue
from worker import conn

redis_q = Queue(connection=conn)
...

这是worker.py:

import os
import redis
from rq import Worker, Queue, Connection
listen = ['high', 'default', 'low']
redis_url = os.getenv('REDISTOGO_URL', 'redis://localhost:6379')
conn = redis.from_url(redis_url)
if __name__ == '__main__':
    with Connection(conn):
        worker = Worker(map(Queue, listen))
        worker.work()

以下是进行后台调用的视图片段:

from upload import *  # where process_upload_file() lives
...
    catchall = {...}
    catchall_string = str(catchall)
    redis_q.enqueue(process_upload_file, catchall_string)
...

upload.py有:

def process_upload_file(catchall_string):
    ...

为什么这不起作用?

(为了额外的功劳,如果有人能给我一个更好的方法将参数作为字符串传递,我们将不胜感激。)

Procfile:

web: gunicorn exim.wsgi -b "0.0.0.0:$PORT"
worker: python worker.py

以下是设置S3存储桶并将heroku指向它后的最新尝试的heroku日志。它无法找到“http://s3.amazonaws.com/mycompany/sample.csv”(此处显示)或“http://mycompany.s3.amazonaws.com/sample.csv”。

    0:46:31+00:00 app[worker.1]: [2013-02-09 00:46] INFO: worker: 
    2013-02-09T00:46:44+00:00 heroku[web.1]: Starting process with command `gunicorn myapp.wsgi -b "0.0.0.0:27999"`
    2013-02-09T00:46:45+00:00 app[web.1]: 2013-02-09 00:46:45 [2] [INFO] Starting gunicorn 0.15.0
    2013-02-09T00:46:45+00:00 app[web.1]: 2013-02-09 00:46:45 [2] [INFO] Listening at: http://0.0.0.0:27999 (2)
    2013-02-09T00:46:45+00:00 app[web.1]: 2013-02-09 00:46:45 [2] [INFO] Using worker: sync
    2013-02-09T00:46:45+00:00 app[web.1]: 2013-02-09 00:46:45 [5] [INFO] Booting worker with pid: 5
    2013-02-09T00:46:50+00:00 heroku[web.1]: State changed from starting to up
    2013-02-09T00:47:15+00:00 heroku[router]: at=info method=GET path=/favicon.ico host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=3ms service=3132ms status=404 bytes=30
    2013-02-09T00:47:15+00:00 heroku[router]: at=info method=GET path=/upl/upload75/ host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=6ms service=980ms status=200 bytes=958
    2013-02-09T00:47:15+00:00 heroku[router]: at=info method=GET path=/favicon.ico host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=2ms service=16ms status=404 bytes=30
    2013-02-09T00:47:21+00:00 heroku[router]: at=info method=POST path=/upl/upload75/ host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=4ms service=244ms status=302 bytes=5
    2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/upl/upload77/ host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=2ms service=162ms status=200 bytes=44024
    2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/css/bootstrap.min.css host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=1ms connect=3ms service=26ms status=200 bytes=95438
    2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/css/style.css host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=2ms service=15ms status=200 bytes=515
    2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/js/jquery-1.7.2.min.js host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=3ms service=17ms status=200 bytes=94840
    2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/js/scripts.js host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=1ms connect=6ms service=5ms status=200 bytes=509
    2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/js/bootstrap.min.js host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=2ms service=7ms status=200 bytes=22323
    2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/snippets/footer.html host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=2ms service=16ms status=200 bytes=537
    2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/img/groovepaper.png host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=1ms service=11ms status=200 bytes=74596
    2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/static/font/fontawesome-webfont.woff host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=3ms service=12ms status=200 bytes=41752
    2013-02-09T00:47:22+00:00 heroku[router]: at=info method=GET path=/favicon.ico host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=1ms service=15ms status=404 bytes=30
    2013-02-09T00:47:37+00:00 heroku[router]: at=info method=POST path=/upl/upload77/ host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=2ms service=182ms status=200 bytes=98
    2013-02-09T00:47:37+00:00 app[worker.1]: [2013-02-09 00:47] INFO: worker: default: upload.process_upload_file("{'server_file': u'/app/myapp/myapp/uploads/d27/org20130208194721.csv', 'client_file': u'aorg.csv', 'myco_id': 27, 'user_email': u'me.myself@mycompany.com', 'ignore_header_line': True, 'content_type': 'org', 'column_choices': {u'org_name': 0, u'addr1': 1, u'country_code': 8}}") (eb79ed7e-cff8-4071-a0c8-b1d4b1bd834f)
    2013-02-09T00:47:37+00:00 heroku[router]: at=info method=GET path=/favicon.ico host=chewy-chiclet-3186.herokuapp.com fwd=71.184.228.19 dyno=web.1 queue=0 wait=0ms connect=1ms service=9ms status=404 bytes=30
    2013-02-09T00:47:42+00:00 app[worker.1]: [2013-02-09 00:47] ERROR: horse: IOError: [Errno 2] No such file or directory: 'http://s3.amazonaws.com/mycompany/sample.csv'
    2013-02-09T00:47:42+00:00 app[worker.1]:     rv = job.perform()
    2013-02-09T00:47:42+00:00 app[worker.1]:   File "/app/refactor.py", line 72, in read_uploaded_file
    2013-02-09T00:47:42+00:00 app[worker.1]:     allrows = csv.reader(open(filepath, 'rU'), dialect='excel')
    2013-02-09T00:47:42+00:00 app[worker.1]: IOError: [Errno 2] No such file or directory: 'http://s3.amazonaws.com/mycompany/sample.csv'
    2013-02-09T00:47:42+00:00 app[worker.1]:   File "/app/refactor.py", line 41, in read_csv
    2013-02-09T00:47:42+00:00 app[worker.1]: [2013-02-09 00:47] WARNING: horse: Moving job to failed queue.
    2013-02-09T00:47:42+00:00 app[worker.1]: Traceback (most recent call last):
    2013-02-09T00:47:42+00:00 app[worker.1]:     output, nrows, error = acceptable_upload_filetypes[ext](filepath, **kwargs)
    2013-02-09T00:47:42+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/rq/worker.py", line 393, in perform_job
    2013-02-09T00:47:42+00:00 app[worker.1]: 
    2013-02-09T00:47:42+00:00 app[worker.1]: [2013-02-09 00:47] DEBUG: horse: Invoking exception handler <bound method Worker.move_to_failed_queue of <rq.worker.Worker object at 0x12cb310>>
    2013-02-09T00:47:42+00:00 app[worker.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/rq/job.py", line 328, in perform
    2013-02-09T00:47:42+00:00 app[worker.1]:     self._result = self.func(*self.args, **self.kwargs)
    2013-02-09T00:47:42+00:00 app[worker.1]:   File "/app/upload.py", line 248, in process_upload_file
    2013-02-09T00:47:42+00:00 app[worker.1]:     all_rows, nrows, error = read_uploaded_file(server_file, maxrows=maxrows)
    2013-02-09T00:47:42+00:00 app[worker.1]: [2013-02-09 00:47] INFO: worker: 
    2013-02-09T00:47:42+00:00 app[worker.1]: [2013-02-09 00:47] INFO: worker: *** Listening on high, default, low...
    (venv)me@host:~/myapp$ 

0 个答案:

没有答案