Pip安装被杀死-内存不足-如何解决?

时间:2019-07-16 13:41:09

标签: python pip

我正在尝试安装spaCy,这是用于自然语言处理的库。

root@vps:~# python3 -m pip install spacy
Collecting spacy
  Using cached https://files.pythonhosted.org/packages/8f/ac/f7a56f8de3bd29384247babb327b7f3c0e3ddef40d1eff8ae993e6d4d2ba/spacy-2.1.6-cp35-cp35m-manylinux1_x86_64.whl
Installing collected packages: spacy
Killed

一段时间后,它被杀死了。我已经在堆栈溢出中寻找答案并尝试过,这是由于内存不足(我的VPS上有1,2 Gb的可用空间,无法扩展)。快速的tail -f /var/log/kern.log给了我Out of memory: Kill process 31729 (python3) score 558 or sacrifice child

我已经尝试添加--no-cache-dir,但没有帮助。手动复制文件时似乎缺少某些内容。有什么方法可以在低RAM服务器上安装此软件包?

编辑:python3 -m pip install spacy -vvv的完整日志

Converted retries value: Retry(total=5, connect=None, read=None, redirect=None)                                                                                                              -> Retry(total=Retry(total=5, connect=None, read=None, redirect=None), connect=N                                                                                                             one, read=None, redirect=None)
Converted retries value: Retry(total=5, connect=None, read=None, redirect=None)                                                                                                              -> Retry(total=Retry(total=5, connect=None, read=None, redirect=None), connect=N                                                                                                             one, read=None, redirect=None)
Collecting spacy
  1 location(s) to search for versions of spacy:
  * https://pypi.python.org/simple/spacy/
  Getting page https://pypi.python.org/simple/spacy/
  Looking up "https://pypi.python.org/simple/spacy/" in the cache
  Returning cached "301 Moved Permanently" response (ignoring date and etag info                                                                                                             rmation)
  Looking up "https://pypi.org/simple/spacy/" in the cache
  Current age based on date: 29458
  Freshness lifetime from max-age: 600
  Freshness lifetime from request max-age: 600
  Starting new HTTPS connection (1): pypi.org
  https://pypi.org:443 "GET /simple/spacy/ HTTP/1.1" 304 0
  Analyzing links from page https://pypi.org/simple/spacy/
    <!-- Found link, skipped linked (hundreds of lines) -->
    Found link https://files.pythonhosted.org/packages/a7/84/f35f44560f0ab9ef23c                                                                                                             4a334a513f2b6c8a95d7b3347ef68a0247da63c08/spacy-2.1.6.tar.gz#sha256=b708bb582597                                                                                                             6220de78ad7e20e827912ca95653738a812cc2e5d194e3060ae8 (from https://pypi.org/simp                                                                                                             le/spacy/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*), version: 2.1                                                                                                             .6
  Using version 2.1.6 (newest of versions: 0.31, 0.32, 0.33, 0.40, 0.51, 0.52, 0                                                                                                             .60, 0.61, 0.62, 0.63, 0.64, 0.65, 0.67, 0.68, 0.70, 0.80, 0.81, 0.82, 0.83, 0.8                                                                                                             4, 0.85, 0.86, 0.87, 0.88, 0.89, 0.90, 0.91, 0.92, 0.93, 0.94, 0.95, 0.97, 0.98,                                                                                                              0.99, 0.100.0, 0.100.1, 0.100.2, 0.100.3, 0.100.4, 0.100.5, 0.100.6, 0.100.7, 0                                                                                                             .101.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.1.0, 1.1.1, 1.1.2, 1.2.0, 1.3.0, 1.                                                                                                             4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.5, 1.8.0, 1.8.1, 1.8.2                                                                                                             , 1.9.0, 1.10.0, 1.10.1, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.0.5, 2.0.6, 2.0.7,                                                                                                              2.0.8, 2.0.9, 2.0.10, 2.0.11, 2.0.12, 2.0.13, 2.0.15, 2.0.16, 2.0.17, 2.0.18, 2                                                                                                             .1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.6)
  Looking up "https://files.pythonhosted.org/packages/8f/ac/f7a56f8de3bd29384247                                                                                                             babb327b7f3c0e3ddef40d1eff8ae993e6d4d2ba/spacy-2.1.6-cp35-cp35m-manylinux1_x86_6                                                                                                             4.whl" in the cache
  Current age based on date: 98439
  Freshness lifetime from max-age: 365000000
  The response is "fresh", returning cached response
  365000000 > 98439
  Using cached https://files.pythonhosted.org/packages/8f/ac/f7a56f8de3bd2938424                                                                                                             7babb327b7f3c0e3ddef40d1eff8ae993e6d4d2ba/spacy-2.1.6-cp35-cp35m-manylinux1_x86_                                                                                                             64.whl
  Downloading from URL https://files.pythonhosted.org/packages/8f/ac/f7a56f8de3b                                                                                                             d29384247babb327b7f3c0e3ddef40d1eff8ae993e6d4d2ba/spacy-2.1.6-cp35-cp35m-manylin                                                                                                             ux1_x86_64.whl#sha256=2441ad9e6195669c13f4035dd77c4206414a40ebb8543c69f3386d984a                                                                                                             bfb942 (from https://pypi.org/simple/spacy/) (requires-python:>=2.7,!=3.0.*,!=3.                                                                                                             1.*,!=3.2.*,!=3.3.*)
Requirement already satisfied: plac<1.0.0,>=0.9.6 in /usr/local/lib/python3.5/di                                                                                                             st-packages (from spacy)
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.5/d                                                                                                             ist-packages (from spacy)
Requirement already satisfied: srsly<1.1.0,>=0.0.6 in /usr/local/lib/python3.5/d                                                                                                             ist-packages (from spacy)
Requirement already satisfied: blis<0.3.0,>=0.2.2 in /usr/local/lib/python3.5/di                                                                                                             st-packages (from spacy)
Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3                                                                                                             .5/dist-packages (from spacy)
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/pytho                                                                                                             n3.5/dist-packages (from spacy)
Requirement already satisfied: wasabi<1.1.0,>=0.2.0 in /usr/local/lib/python3.5/                                                                                                             dist-packages (from spacy)
Requirement already satisfied: thinc<7.1.0,>=7.0.8 in /usr/local/lib/python3.5/d                                                                                                             ist-packages (from spacy)
Requirement already satisfied: preshed<2.1.0,>=2.0.1 in /usr/local/lib/python3.5                                                                                                             /dist-packages (from spacy)
Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.5/dist-pa                                                                                                             ckages (from spacy)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/l                                                                                                             ocal/lib/python3.5/dist-packages (from requests<3.0.0,>=2.13.0->spacy)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.5/dist-p                                                                                                             ackages (from requests<3.0.0,>=2.13.0->spacy)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.5/di                                                                                                             st-packages (from requests<3.0.0,>=2.13.0->spacy)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.5                                                                                                             /dist-packages (from requests<3.0.0,>=2.13.0->spacy)
Requirement already satisfied: tqdm<5.0.0,>=4.10.0 in /usr/local/lib/python3.5/d                                                                                                             ist-packages (from thinc<7.1.0,>=7.0.8->spacy)
Installing collected packages: spacy
Killed

3 个答案:

答案 0 :(得分:2)

这已经在github上报道了

https://github.com/explosion/spaCy/issues/3396

下面是一些有关如何解决此问题的好线索

fork() failing with Out of memory error

Memory error while using pip install Matplotlib

out of memory issue in installing packages on Ubuntu server

https://chirale.org/2017/01/15/memory-error-on-pip-install-solved/

摘要中,以下是两个选项:

  1. 尝试使用--no-cache-dir,很遗憾,该方法对您没有帮助

  2. 增加交换空间

$ create swap file of 512 MB
dd if=/dev/zero of=/swapfile bs=1024 count=524288

$ modify permissions
chown root:root /swapfile
chmod 0600 /swapfile

$ setup swap area
mkswap /swapfile

$ turn swap on
swapon /swapfile

答案 1 :(得分:1)

尝试将交换空间添加到您的VPS。这样您就可以将可用磁盘空间用作内存

答案 2 :(得分:0)

不确定这是否行得通,但您尝试从源代码编译吗?该脚本位于Spacy's pypi page。不知道这是否会比常规的pip安装更好地管理RAM约束,但这是我会尝试的方法。

# make sure you are using the latest pip
python -m pip install -U pip
git clone https://github.com/explosion/spaCy
cd spaCy

python -m venv .env
source .env/bin/activate
export PYTHONPATH=`pwd`
pip install -r requirements.txt
python setup.py build_ext --inplace