Question

我尝试在Ubuntu上运行带有crontab的python scrapy爬虫，但是我收到了以下错误消息：

Traceback (most recent call last):
  File "/usr/bin/scrapy", line 9, in <module>
    load_entry_point('Scrapy==1.0.3', 'console_scripts', 'scrapy')()
  File "/usr/lib/python2.7/dist-packages/scrapy/cmdline.py", line 142, in execu$
    cmd.crawler_process = CrawlerProcess(settings)
  File "/usr/lib/python2.7/dist-packages/scrapy/crawler.py", line 209, in __ini$
    super(CrawlerProcess, self).__init__(settings)
  File "/usr/lib/python2.7/dist-packages/scrapy/crawler.py", line 115, in __ini$
    self.spider_loader = _get_spider_loader(settings)
  File "/usr/lib/python2.7/dist-packages/scrapy/crawler.py", line 296, in _get_$
    return loader_cls.from_settings(settings.frozencopy())
  File "/usr/lib/python2.7/dist-packages/scrapy/spiderloader.py", line 30, in f$
    return cls(settings)
  File "/usr/lib/python2.7/dist-packages/scrapy/spiderloader.py", line 21, in _$
    for module in walk_modules(name):
  File "/usr/lib/python2.7/dist-packages/scrapy/utils/misc.py", line 71, in wal$
    submod = import_module(fullpath)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/home/kebodev/scrapy/qgtest2/qgtest2/spiders/jsonspider.py", line 5, in <module>
    import cx_Oracle
ImportError: libclntsh.so.11.1: cannot open shared object file: No such file or directory

我用root用户编辑〜/ .bashrc并添加以下行：

export ORACLE_HOME=/u01/app/oracle/product/11.2.0/xe
export ORACLE_SID=XE
export NLS_LANG=`$ORACLE_HOME/bin/nls_lang.sh`
export ORACLE_BASE=/u01/app/oracle
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH
export PATH=$ORACLE_HOME/bin:$PATH

我的libclntsh.so.11.1位于：/u01/app/oracle/product/11.2.0/xe/lib

如果我尝试从终端运行我的python scrapy crawler，它正在运行，如果我尝试在python shell中导入cx_Oracle，它也可以工作，但是使用crontab它不行..

这就是我的cron工作线的样子：

* * * * * root /etc/listarunner.sh >> /home/kebodev/scrapy/qgtest2/etcronlog1.log 2>&1

这是我的listarunner.sh文件：

#!/bin/bash
cd /home/kebodev/scrapy/qgtest2
PATH=$PATH:/usr/local/bin
export PATH
scrapy crawl jsontst

有人能帮助我吗？

谢谢！

Answer 1

我可以尝试一些选项：

1）在listarunner.sh中设置环境变量而不是〜/ .bashrc

2）使用/etc/ld.so.conf.d中的文件来设置LD_LIBRARY_PATH

3）重建cx_Oracle，首先在构建之前将环境变量FORCE_RPATH设置为任何值。

希望其中一个会帮助你！

Python cx_Oracle错误

1 个答案: