在AWS Jupyter EMR中找不到python软件包

时间:2020-08-03 13:12:33

标签: python jupyter-notebook jupyter conda amazon-emr

我是AWS EMR Jupyter环境的新手,并被困在python软件包的安装中。我现在已经设置了一个单节点EMR集群,并通过对节点进行SSH将此程序包looker_sdk安装在docker容器中。我可以看到它已安装,并可以使用iPython / python shell看到它,并且可以导入它。但是当我尝试从Jupyter Notebook导入时,它仍然失败。

root@jupyterhub:~# pip freeze | grep looker
WARNING: The directory '/home/jovyan/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
looker-sdk==0.1.3b15

但是在Jupyter中:

print(sys.path)
import looker_sdk

['/home/notebook/work', '/opt/conda/lib/python37.zip', '/opt/conda/lib/python3.7', '/opt/conda/lib/python3.7/lib-dynload', '', '/opt/conda/lib/python3.7/site-packages', '/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg', '/opt/conda/lib/python3.7/site-packages/IPython/extensions', '/home/notebook/.ipython']
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-13-f7a797152935> in <module>
      1 print(sys.path)
----> 2 import looker_sdk

ModuleNotFoundError: No module named 'looker_sdk'

此外,目录列表给出了不同的结果,拥有目录的用户也不同。

从Docker容器中的Shell中获取

root@jupyterhub:~# ls -ltr /opt/conda/
total 56
-rw-rw-r--  1 jovyan users 11245 Mar  3 01:40 LICENSE.txt
drwsrwsr-x  1 jovyan users    21 Mar 17 21:11 x86_64-conda_cos6-linux-gnu
drwsrwsr-x  3 jovyan users    22 Mar 17 21:11 shell
drwsrwsr-x  2 jovyan users    19 Mar 17 21:16 condabin
drwsrwsr-x  2 jovyan users    30 Mar 17 21:16 compiler_compat
drwsrwsr-x  2 jovyan users   203 Mar 17 21:21 sbin
drwsrwsr-x  1 jovyan users   146 Apr  8 21:36 ssl
drwxr-sr-x  3 jovyan users    19 Apr  8 21:37 var
drwxr-sr-x  2 jovyan users   175 Apr  8 21:37 resources
drwxr-sr-x 26 jovyan users  4096 Apr  8 21:37 qml
drwxr-sr-x  2 jovyan users   252 Apr  8 21:37 phrasebooks
drwxr-sr-x 75 jovyan users  4096 Apr  8 21:37 mkspecs
drwxr-sr-x  3 jovyan users    86 Apr  8 21:37 libexec
drwsrwsr-x  1 jovyan users    23 Apr  8 21:37 lib
drwsrwsr-x  1 jovyan users  4096 Apr  8 21:37 include
drwsrwsr-x  1 jovyan users    21 Apr  8 21:37 etc
drwxr-sr-x  3 jovyan users    20 Apr  8 21:37 doc
drwxr-sr-x  3 jovyan users 12288 Apr  8 21:37 translations
drwsrwsr-x  1 jovyan users    21 Apr  8 21:37 share
drwxr-sr-x  3 jovyan users    17 Apr  8 21:37 qsci
drwxr-sr-x 30 jovyan users  4096 Apr  8 21:37 plugins
drwxr-sr-x  3 jovyan users    18 Apr  8 21:37 man
drwsrwsr-x  1 jovyan users    21 Apr  8 21:38 conda-meta
drwsrwsr-x  1 jovyan users   212 Apr  8 21:38 bin
drwsrwsr-x  1 jovyan users    55 Aug  3 12:16 envs
drwxr-sr-x  1 jovyan users  8192 Aug  3 12:56 pkgs

来自Jupyter:

!ls -ltr /opt/conda
total 124
-rw-rw-r--  1 notebook users 11245 Mar  3 01:41 LICENSE.txt
drwsrwsr-x  1 notebook users  4096 Apr 30 01:44 x86_64-conda_cos6-linux-gnu
drwsrwsr-x  3 notebook users  4096 Apr 30 01:44 man
drwsrwsr-x  3 notebook users  4096 Apr 30 01:44 var
drwsrwsr-x  3 notebook users  4096 Apr 30 01:44 doc
drwsrwsr-x  3 notebook users  4096 Apr 30 01:44 shell
drwsrwsr-x  2 notebook users  4096 Apr 30 01:44 envs
drwsrwsr-x  1 notebook users  4096 Apr 30 01:46 etc
drwsrwsr-x  3 notebook users  4096 Apr 30 01:46 qsci
drwsrwsr-x 30 notebook users  4096 Apr 30 01:46 plugins
drwsrwsr-x  2 notebook users  4096 Apr 30 01:47 condabin
drwsrwsr-x  3 notebook users 12288 Apr 30 01:47 translations
drwsrwsr-x  2 notebook users  4096 Apr 30 01:47 resources
drwsrwsr-x 26 notebook users  4096 Apr 30 01:47 qml
drwsrwsr-x  2 notebook users  4096 Apr 30 01:47 phrasebooks
drwsrwsr-x 75 notebook users  4096 Apr 30 01:47 mkspecs
drwsrwsr-x  4 notebook users  4096 Apr 30 01:47 libexec
drwsrwsr-x  2 notebook users  4096 Apr 30 01:47 compiler_compat
drwsrwsr-x  1 root     users  4096 Apr 30 01:50 locks
drwsrwsr-x  1 notebook users  4096 Apr 30 01:53 ssl
drwsrwsr-x  1 notebook users  4096 Apr 30 01:54 share
drwsrwsr-x  1 notebook users  4096 Apr 30 01:54 sbin
drwsrwsr-x  1 notebook users  4096 Apr 30 01:54 lib
drwsrwsr-x  1 notebook users  4096 Apr 30 01:54 include
drwsrwsr-x  1 notebook users  4096 Apr 30 01:57 bin
drwsrwsr-x  1 notebook users  4096 Apr 30 02:02 pkgs
drwsrwsr-x  1 notebook users  4096 Apr 30 02:02 conda-meta

我试图研究并阅读有关此内容的信息,但是找不到任何可以解释其工作原理的信息。请在这里帮助。

更新:如果直接进入JupyterHub控制台,则会看到使用了用户“ jovyan”。但是,如果我们通过附加集群从EMR控制台中打开笔记本,则会使用用户“笔记本”,并且看不到我们为安装内核所做的任何更改。我将尽快破解答案。

0 个答案:

没有答案