我正在尝试安装spaCy,这是用于自然语言处理的库。
root@vps:~# python3 -m pip install spacy
Collecting spacy
Using cached https://files.pythonhosted.org/packages/8f/ac/f7a56f8de3bd29384247babb327b7f3c0e3ddef40d1eff8ae993e6d4d2ba/spacy-2.1.6-cp35-cp35m-manylinux1_x86_64.whl
Installing collected packages: spacy
Killed
一段时间后,它被杀死了。我已经在堆栈溢出中寻找答案并尝试过,这是由于内存不足(我的VPS上有1,2 Gb的可用空间,无法扩展)。快速的tail -f /var/log/kern.log
给了我Out of memory: Kill process 31729 (python3) score 558 or sacrifice child
。
我已经尝试添加--no-cache-dir
,但没有帮助。手动复制文件时似乎缺少某些内容。有什么方法可以在低RAM服务器上安装此软件包?
编辑:python3 -m pip install spacy -vvv
的完整日志
Converted retries value: Retry(total=5, connect=None, read=None, redirect=None) -> Retry(total=Retry(total=5, connect=None, read=None, redirect=None), connect=N one, read=None, redirect=None)
Converted retries value: Retry(total=5, connect=None, read=None, redirect=None) -> Retry(total=Retry(total=5, connect=None, read=None, redirect=None), connect=N one, read=None, redirect=None)
Collecting spacy
1 location(s) to search for versions of spacy:
* https://pypi.python.org/simple/spacy/
Getting page https://pypi.python.org/simple/spacy/
Looking up "https://pypi.python.org/simple/spacy/" in the cache
Returning cached "301 Moved Permanently" response (ignoring date and etag info rmation)
Looking up "https://pypi.org/simple/spacy/" in the cache
Current age based on date: 29458
Freshness lifetime from max-age: 600
Freshness lifetime from request max-age: 600
Starting new HTTPS connection (1): pypi.org
https://pypi.org:443 "GET /simple/spacy/ HTTP/1.1" 304 0
Analyzing links from page https://pypi.org/simple/spacy/
<!-- Found link, skipped linked (hundreds of lines) -->
Found link https://files.pythonhosted.org/packages/a7/84/f35f44560f0ab9ef23c 4a334a513f2b6c8a95d7b3347ef68a0247da63c08/spacy-2.1.6.tar.gz#sha256=b708bb582597 6220de78ad7e20e827912ca95653738a812cc2e5d194e3060ae8 (from https://pypi.org/simp le/spacy/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*), version: 2.1 .6
Using version 2.1.6 (newest of versions: 0.31, 0.32, 0.33, 0.40, 0.51, 0.52, 0 .60, 0.61, 0.62, 0.63, 0.64, 0.65, 0.67, 0.68, 0.70, 0.80, 0.81, 0.82, 0.83, 0.8 4, 0.85, 0.86, 0.87, 0.88, 0.89, 0.90, 0.91, 0.92, 0.93, 0.94, 0.95, 0.97, 0.98, 0.99, 0.100.0, 0.100.1, 0.100.2, 0.100.3, 0.100.4, 0.100.5, 0.100.6, 0.100.7, 0 .101.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.1.0, 1.1.1, 1.1.2, 1.2.0, 1.3.0, 1. 4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.5, 1.8.0, 1.8.1, 1.8.2 , 1.9.0, 1.10.0, 1.10.1, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.0.5, 2.0.6, 2.0.7, 2.0.8, 2.0.9, 2.0.10, 2.0.11, 2.0.12, 2.0.13, 2.0.15, 2.0.16, 2.0.17, 2.0.18, 2 .1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.6)
Looking up "https://files.pythonhosted.org/packages/8f/ac/f7a56f8de3bd29384247 babb327b7f3c0e3ddef40d1eff8ae993e6d4d2ba/spacy-2.1.6-cp35-cp35m-manylinux1_x86_6 4.whl" in the cache
Current age based on date: 98439
Freshness lifetime from max-age: 365000000
The response is "fresh", returning cached response
365000000 > 98439
Using cached https://files.pythonhosted.org/packages/8f/ac/f7a56f8de3bd2938424 7babb327b7f3c0e3ddef40d1eff8ae993e6d4d2ba/spacy-2.1.6-cp35-cp35m-manylinux1_x86_ 64.whl
Downloading from URL https://files.pythonhosted.org/packages/8f/ac/f7a56f8de3b d29384247babb327b7f3c0e3ddef40d1eff8ae993e6d4d2ba/spacy-2.1.6-cp35-cp35m-manylin ux1_x86_64.whl#sha256=2441ad9e6195669c13f4035dd77c4206414a40ebb8543c69f3386d984a bfb942 (from https://pypi.org/simple/spacy/) (requires-python:>=2.7,!=3.0.*,!=3. 1.*,!=3.2.*,!=3.3.*)
Requirement already satisfied: plac<1.0.0,>=0.9.6 in /usr/local/lib/python3.5/di st-packages (from spacy)
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.5/d ist-packages (from spacy)
Requirement already satisfied: srsly<1.1.0,>=0.0.6 in /usr/local/lib/python3.5/d ist-packages (from spacy)
Requirement already satisfied: blis<0.3.0,>=0.2.2 in /usr/local/lib/python3.5/di st-packages (from spacy)
Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3 .5/dist-packages (from spacy)
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/pytho n3.5/dist-packages (from spacy)
Requirement already satisfied: wasabi<1.1.0,>=0.2.0 in /usr/local/lib/python3.5/ dist-packages (from spacy)
Requirement already satisfied: thinc<7.1.0,>=7.0.8 in /usr/local/lib/python3.5/d ist-packages (from spacy)
Requirement already satisfied: preshed<2.1.0,>=2.0.1 in /usr/local/lib/python3.5 /dist-packages (from spacy)
Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.5/dist-pa ckages (from spacy)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/l ocal/lib/python3.5/dist-packages (from requests<3.0.0,>=2.13.0->spacy)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.5/dist-p ackages (from requests<3.0.0,>=2.13.0->spacy)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.5/di st-packages (from requests<3.0.0,>=2.13.0->spacy)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.5 /dist-packages (from requests<3.0.0,>=2.13.0->spacy)
Requirement already satisfied: tqdm<5.0.0,>=4.10.0 in /usr/local/lib/python3.5/d ist-packages (from thinc<7.1.0,>=7.0.8->spacy)
Installing collected packages: spacy
Killed
答案 0 :(得分:2)
这已经在github上报道了
https://github.com/explosion/spaCy/issues/3396
下面是一些有关如何解决此问题的好线索
fork() failing with Out of memory error
Memory error while using pip install Matplotlib
out of memory issue in installing packages on Ubuntu server
https://chirale.org/2017/01/15/memory-error-on-pip-install-solved/
摘要中,以下是两个选项:
尝试使用--no-cache-dir
,很遗憾,该方法对您没有帮助
增加交换空间
$ create swap file of 512 MB
dd if=/dev/zero of=/swapfile bs=1024 count=524288
$ modify permissions
chown root:root /swapfile
chmod 0600 /swapfile
$ setup swap area
mkswap /swapfile
$ turn swap on
swapon /swapfile
答案 1 :(得分:1)
尝试将交换空间添加到您的VPS。这样您就可以将可用磁盘空间用作内存
答案 2 :(得分:0)
不确定这是否行得通,但您尝试从源代码编译吗?该脚本位于Spacy's pypi page。不知道这是否会比常规的pip安装更好地管理RAM约束,但这是我会尝试的方法。
# make sure you are using the latest pip
python -m pip install -U pip
git clone https://github.com/explosion/spaCy
cd spaCy
python -m venv .env
source .env/bin/activate
export PYTHONPATH=`pwd`
pip install -r requirements.txt
python setup.py build_ext --inplace