Question

我知道scrapy应首先安装w3lib，所以我先安装w3lib，但是当我在python ide中导入scrapy时，程序崩溃了。错误：

creating Twisted.egg-info

writing requirements to Twisted.egg-info\requires.txt

writing Twisted.egg-info\PKG-INFO

writing top-level names to Twisted.egg-info\top_level.txt

writing dependency_links to Twisted.egg-info\dependency_links.txt

writing manifest file 'Twisted.egg-info\SOURCES.txt'

warning: manifest_maker: standard file '-c' not found



reading manifest file 'Twisted.egg-info\SOURCES.txt'

writing manifest file 'Twisted.egg-info\SOURCES.txt'

copying twisted\internet\_sigchld.c -> build\lib.win-amd64-2.7\twisted\internet

creating build\lib.win-amd64-2.7\twisted\internet\iocpreactor\iocpsupport

copying twisted\internet/iocpreactor/iocpsupport\iocpsupport.c -> build\lib.win-amd64-2.7\twisted\internet/iocpreactor/i
ocpsupport

copying twisted\internet/iocpreactor/iocpsupport\winsock_pointers.c -> build\lib.win-amd64-2.7\twisted\internet/iocpreac
tor/iocpsupport

copying twisted\python\_epoll.c -> build\lib.win-amd64-2.7\twisted\python

copying twisted\python\_initgroups.c -> build\lib.win-amd64-2.7\twisted\python

copying twisted\python\sendmsg.c -> build\lib.win-amd64-2.7\twisted\python

copying twisted\runner\portmap.c -> build\lib.win-amd64-2.7\twisted\runner

copying twisted\test\raiser.c -> build\lib.win-amd64-2.7\twisted\test

running build_ext

怎么了？

Answer 1

确保已安装Twisted，pyOpenSSL和pycrypto。这些是我在ubuntu上安装scrapy的步骤。 1.install gcc和lxml：

sudo apt-get install python-dev
sudo apt-get install libevent-dev
sudo apt-get install libxml2 libxml2-dev
apt-get install libxml2-dev libxslt-dev
apt-get install python-lxml

2.install twisted：

sudo apt-get install python-twisted python-libxml2 python-simplejson

sudo apt-get install build-essential libssl-dev libffi-dev python-dev

3.install pyOpenSSL：

wget http://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.13.tar.gz
tar -zxvf pyOpenSSL-0.13.tar.gz
cd pyOpenSSL-0.13
sudo python setup.py install

4.install pycrypto

wget http://pypi.python.org/packages/source/p/pycrypto/pycrypto-2.5.tar.gz
tar -zxvf pycrypto-2.5.tar.gz
cd pycrypto-2.5
sudo python setup.py install

5.install easy_install :(如果你没有easy_install）

wget http://peak.telecommunity.com/dist/ez_setup.py
python ez_setup.py

6.install w3lib

sudo easy_install -U w3lib

7.install scrapy

sudo easy_install Scrapy

如果你想了解多少，请转到我的博客。

Answer 2

这就是我在ubuntu上安装scrapy的方法：

sudo apt-get update
sudo apt-get install python-pip build-essential python-dev libxslt-dev libxml2-dev   
sudo -H pip install Scrapy
scrapy version

解决我的问题的重要事项是sudo -H pip install Scrapy，特别是-H标志。

我也从终端退出并启动了一个新终端，以确保正确设置所有环境变量

Answer 3

首先安装系统依赖项 sudo apt-get install -y \ python-dev python-pip python-setuptools \ libffi-dev libxml2-dev libxslt1-dev \ libtiff5-dev libjpeg62-turbo-dev zlib1g-dev libfreetype6-dev \ liblcms2-dev libwebp-dev tcl8.5-dev tk8.5-dev python-tk

然后在requirements.txt

中添加以下内容

lxml
pyOpenSSL
Scrapy
Pillow

最后pip install -r requirements.txt

您还可以浏览gist.github.com以解决最新的依赖项问题。我正在使用docker在一个单独的容器中设置scrapy deps。我为我的需要创建了一个here

如何在ubuntu上安装scrapy？

3 个答案: