Question

执行时

curl http://localhost:6800/schedule.json -d project=default -d spider=spider1

我去了scrapyd控制台，我看到了例外：

[-] Unhandled Error
Traceback (most recent call last):
  File "/home/scrapyV1/venv/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/home/scrapyV1/venv/lib/python2.7/site-packages/twisted/internet/defer.py", line 1274, in unwindGenerator
    return _inlineCallbacks(None, gen, Deferred())
  File "/home/scrapyV1/venv/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/home/scrapyV1/venv/lib/python2.7/site-packages/scrapyd/poller.py", line 23, in poll
    msg = yield maybeDeferred(q.pop)
--- <exception caught here> ---
  File "/home/scrapyV1/venv/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/home/scrapyV1/venv/lib/python2.7/site-packages/scrapyd/spiderqueue.py", line 21, in pop
    return self.q.pop()
  File "/home/scrapyV1/venv/lib/python2.7/site-packages/scrapyd/sqlite.py", line 119, in pop
    return self.decode(msg)
  File "/home/scrapyV1/venv/lib/python2.7/site-packages/scrapyd/sqlite.py", line 170, in decode
    return json.loads(text)
  File "/usr/local/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python2.7/json/decoder.py", line 382, in raw_decode
    obj, end = self.scan_once(s, idx)
exceptions.ValueError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)nter code here

之后我无法运行下一个Spider，我得到了

{"status": "error", "message": "No JSON object could be decoded", "node_name": "spider-test2"}

如何确定与错误相关的内容？

Answer 1

问题出在我的Centos6.5中安装了sqlite3。决定：需要添加

self.conn.text_factory = bytes

在scrapyd/sqlite.py b/scrapyd/sqlite.py

中创建连接后，

到SqliteDict和SqlitePriorityQueue类

scrapyd异常期望用双引号括起来的属性名称

1 个答案: