我是AWS EMR Jupyter环境的新手,并被困在python软件包的安装中。我现在已经设置了一个单节点EMR集群,并通过对节点进行SSH将此程序包looker_sdk安装在docker容器中。我可以看到它已安装,并可以使用iPython / python shell看到它,并且可以导入它。但是当我尝试从Jupyter Notebook导入时,它仍然失败。
root@jupyterhub:~# pip freeze | grep looker
WARNING: The directory '/home/jovyan/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
looker-sdk==0.1.3b15
但是在Jupyter中:
print(sys.path)
import looker_sdk
['/home/notebook/work', '/opt/conda/lib/python37.zip', '/opt/conda/lib/python3.7', '/opt/conda/lib/python3.7/lib-dynload', '', '/opt/conda/lib/python3.7/site-packages', '/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg', '/opt/conda/lib/python3.7/site-packages/IPython/extensions', '/home/notebook/.ipython']
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-13-f7a797152935> in <module>
1 print(sys.path)
----> 2 import looker_sdk
ModuleNotFoundError: No module named 'looker_sdk'
此外,目录列表给出了不同的结果,拥有目录的用户也不同。
从Docker容器中的Shell中获取
:root@jupyterhub:~# ls -ltr /opt/conda/
total 56
-rw-rw-r-- 1 jovyan users 11245 Mar 3 01:40 LICENSE.txt
drwsrwsr-x 1 jovyan users 21 Mar 17 21:11 x86_64-conda_cos6-linux-gnu
drwsrwsr-x 3 jovyan users 22 Mar 17 21:11 shell
drwsrwsr-x 2 jovyan users 19 Mar 17 21:16 condabin
drwsrwsr-x 2 jovyan users 30 Mar 17 21:16 compiler_compat
drwsrwsr-x 2 jovyan users 203 Mar 17 21:21 sbin
drwsrwsr-x 1 jovyan users 146 Apr 8 21:36 ssl
drwxr-sr-x 3 jovyan users 19 Apr 8 21:37 var
drwxr-sr-x 2 jovyan users 175 Apr 8 21:37 resources
drwxr-sr-x 26 jovyan users 4096 Apr 8 21:37 qml
drwxr-sr-x 2 jovyan users 252 Apr 8 21:37 phrasebooks
drwxr-sr-x 75 jovyan users 4096 Apr 8 21:37 mkspecs
drwxr-sr-x 3 jovyan users 86 Apr 8 21:37 libexec
drwsrwsr-x 1 jovyan users 23 Apr 8 21:37 lib
drwsrwsr-x 1 jovyan users 4096 Apr 8 21:37 include
drwsrwsr-x 1 jovyan users 21 Apr 8 21:37 etc
drwxr-sr-x 3 jovyan users 20 Apr 8 21:37 doc
drwxr-sr-x 3 jovyan users 12288 Apr 8 21:37 translations
drwsrwsr-x 1 jovyan users 21 Apr 8 21:37 share
drwxr-sr-x 3 jovyan users 17 Apr 8 21:37 qsci
drwxr-sr-x 30 jovyan users 4096 Apr 8 21:37 plugins
drwxr-sr-x 3 jovyan users 18 Apr 8 21:37 man
drwsrwsr-x 1 jovyan users 21 Apr 8 21:38 conda-meta
drwsrwsr-x 1 jovyan users 212 Apr 8 21:38 bin
drwsrwsr-x 1 jovyan users 55 Aug 3 12:16 envs
drwxr-sr-x 1 jovyan users 8192 Aug 3 12:56 pkgs
来自Jupyter:
!ls -ltr /opt/conda
total 124
-rw-rw-r-- 1 notebook users 11245 Mar 3 01:41 LICENSE.txt
drwsrwsr-x 1 notebook users 4096 Apr 30 01:44 x86_64-conda_cos6-linux-gnu
drwsrwsr-x 3 notebook users 4096 Apr 30 01:44 man
drwsrwsr-x 3 notebook users 4096 Apr 30 01:44 var
drwsrwsr-x 3 notebook users 4096 Apr 30 01:44 doc
drwsrwsr-x 3 notebook users 4096 Apr 30 01:44 shell
drwsrwsr-x 2 notebook users 4096 Apr 30 01:44 envs
drwsrwsr-x 1 notebook users 4096 Apr 30 01:46 etc
drwsrwsr-x 3 notebook users 4096 Apr 30 01:46 qsci
drwsrwsr-x 30 notebook users 4096 Apr 30 01:46 plugins
drwsrwsr-x 2 notebook users 4096 Apr 30 01:47 condabin
drwsrwsr-x 3 notebook users 12288 Apr 30 01:47 translations
drwsrwsr-x 2 notebook users 4096 Apr 30 01:47 resources
drwsrwsr-x 26 notebook users 4096 Apr 30 01:47 qml
drwsrwsr-x 2 notebook users 4096 Apr 30 01:47 phrasebooks
drwsrwsr-x 75 notebook users 4096 Apr 30 01:47 mkspecs
drwsrwsr-x 4 notebook users 4096 Apr 30 01:47 libexec
drwsrwsr-x 2 notebook users 4096 Apr 30 01:47 compiler_compat
drwsrwsr-x 1 root users 4096 Apr 30 01:50 locks
drwsrwsr-x 1 notebook users 4096 Apr 30 01:53 ssl
drwsrwsr-x 1 notebook users 4096 Apr 30 01:54 share
drwsrwsr-x 1 notebook users 4096 Apr 30 01:54 sbin
drwsrwsr-x 1 notebook users 4096 Apr 30 01:54 lib
drwsrwsr-x 1 notebook users 4096 Apr 30 01:54 include
drwsrwsr-x 1 notebook users 4096 Apr 30 01:57 bin
drwsrwsr-x 1 notebook users 4096 Apr 30 02:02 pkgs
drwsrwsr-x 1 notebook users 4096 Apr 30 02:02 conda-meta
我试图研究并阅读有关此内容的信息,但是找不到任何可以解释其工作原理的信息。请在这里帮助。
更新:如果直接进入JupyterHub控制台,则会看到使用了用户“ jovyan”。但是,如果我们通过附加集群从EMR控制台中打开笔记本,则会使用用户“笔记本”,并且看不到我们为安装内核所做的任何更改。我将尽快破解答案。